Sep 05 02:17:58.898820 ip-10-0-135-239 systemd[1]: Starting Kubernetes Kubelet...
Sep 05 02:17:59.309138 ip-10-0-135-239 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:59.309138 ip-10-0-135-239 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 05 02:17:59.309138 ip-10-0-135-239 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:59.309138 ip-10-0-135-239 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 05 02:17:59.309138 ip-10-0-135-239 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:59.309844 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.309199    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 05 02:17:59.315637 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315613    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:59.315637 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315635    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:59.315637 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315640    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315643    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315646    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315650    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315653    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315655    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315658    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315661    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315664    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315669    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315672    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315674    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315677    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315686    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315689    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315692    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315695    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315697    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315700    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315703    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:59.315745 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315705    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315708    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315710    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315713    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315715    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315718    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315723    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315727    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315730    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315733    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315736    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315739    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315742    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315744    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315748    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315752    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315754    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315757    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315760    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:59.316242 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315762    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315765    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315768    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315770    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315773    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315775    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315778    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315781    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315784    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315787    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315789    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315792    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315795    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315797    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315800    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315802    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315805    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315808    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315810    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315813    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:59.316714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315817    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315819    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315822    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315824    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315827    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315830    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315832    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315836    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315840    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315843    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315845    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315848    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315851    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315853    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315856    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315858    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315861    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315863    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315866    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:59.317209 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315868    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315873    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315876    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315879    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315882    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315884    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315887    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315891    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315894    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315897    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315900    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315903    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315906    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315909    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315912    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315915    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315917    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315920    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.315923    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:59.317678 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316657    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316665    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316668    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316671    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316674    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316677    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316679    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316682    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316686    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316690    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316693    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316696    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316699    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316702    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316705    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316708    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316712    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316716    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316719    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:59.318181 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316722    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316726    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316730    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316733    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316736    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316739    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316741    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316744    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316747    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316749    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316752    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316755    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316759    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316761    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316765    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316767    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316770    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316772    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316775    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316778    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:59.318690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316780    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316783    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316785    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316788    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316790    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316793    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316796    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316798    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316800    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316803    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316806    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316809    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316812    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316815    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316817    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316820    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316823    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316825    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316828    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316830    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:59.319223 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316833    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316835    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316838    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316840    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316843    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316845    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316848    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316851    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316853    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316857    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316860    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316862    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316864    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316867    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316869    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316872    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316874    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316877    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316879    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316882    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:59.319723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316884    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316887    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316890    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316893    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316895    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316898    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316900    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316903    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316905    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316908    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316910    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316913    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316915    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316918    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316921    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316923    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316926    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316929    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316932    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.316935    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:59.320215 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317353    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317358    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317361    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317365    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317368    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317371    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317374    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317377    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317380    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317382    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317385    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317387    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317390    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317392    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317396    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317400    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317404    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317406    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317410    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317413    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:59.320723 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317415    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317418    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317421    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317423    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317426    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317429    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317431    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317434    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317436    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317439    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317442    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317444    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317448    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317452    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317455    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317457    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317460    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317462    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317465    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317467    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:59.321211 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317470    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317472    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317475    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317478    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317481    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317483    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317486    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317489    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317492    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317494    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317497    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317499    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317503    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317506    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317509    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317511    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317514    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317517    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317520    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:59.321714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317522    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317525    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317527    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317547    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317553    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317556    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317560    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317564    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317568    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317572    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317574    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317577    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317579    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317582    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317584    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317587    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317589    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317592    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317594    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317597    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:59.322169 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317600    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317604    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317606    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317610    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317613    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317615    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317618    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317621    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317626    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317629    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317631    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317634    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317636    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317639    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317642    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317644    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317647    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317650    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317652    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.317655    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:59.322651 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.317733    2555 flags.go:64] FLAG: --address="0.0.0.0"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.317778    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318424    2555 flags.go:64] FLAG: --anonymous-auth="true"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318429    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318437    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318440    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318444    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318449    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318452    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318455    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318459    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318463    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318467    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318470    2555 flags.go:64] FLAG: --cgroup-root=""
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318474    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318477    2555 flags.go:64] FLAG: --client-ca-file=""
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318480    2555 flags.go:64] FLAG: --cloud-provider="external"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318483    2555 flags.go:64] FLAG: --cluster-dns="[]"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318487    2555 flags.go:64] FLAG: --cluster-domain=""
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318490    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318496    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318499    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318503    2555 flags.go:64] FLAG: --container-log-max-files="5"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318506    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 05 02:17:59.323145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318509    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318513    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318516    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318519    2555 flags.go:64] FLAG: --contention-profiling="false"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318523    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318527    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318544    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318547    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318552    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318555    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318558    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318561    2555 flags.go:64] FLAG: --enable-load-reader="false"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318564    2555 flags.go:64] FLAG: --enable-server="true"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318567    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318576    2555 flags.go:64] FLAG: --event-burst="100"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318588    2555 flags.go:64] FLAG: --event-qps="50"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318591    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318595    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318598    2555 flags.go:64] FLAG: --eviction-hard=""
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318602    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318605    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318608    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318611    2555 flags.go:64] FLAG: --eviction-soft=""
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318615    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318618    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 05 02:17:59.323744 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318622    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318624    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318627    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318632    2555 flags.go:64] FLAG: --fail-swap-on="true"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318635    2555 flags.go:64] FLAG: --feature-gates=""
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318639    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318642    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318645    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318648    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318652    2555 flags.go:64] FLAG: --healthz-port="10248"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318655    2555 flags.go:64] FLAG: --help="false"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318658    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318661    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318664    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318667    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318671    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318675    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318678    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318681    2555 flags.go:64] FLAG: --image-service-endpoint=""
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318684    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318687    2555 flags.go:64] FLAG: --kube-api-burst="100"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318690    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318693    2555 flags.go:64] FLAG: --kube-api-qps="50"
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318696    2555 flags.go:64] FLAG: --kube-reserved=""
Sep 05 02:17:59.324520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318699    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318702    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318706    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318709    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318712    2555 flags.go:64] FLAG: --lock-file=""
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318715    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318718    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318722    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318728    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318731    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318734    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318739    2555 flags.go:64] FLAG: --logging-format="text"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318742    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318746    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318748    2555 flags.go:64] FLAG: --manifest-url=""
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318751    2555 flags.go:64] FLAG: --manifest-url-header=""
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318756    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318759    2555 flags.go:64] FLAG: --max-open-files="1000000"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318764    2555 flags.go:64] FLAG: --max-pods="110"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318767    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318770    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318773    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318777    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318780    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318783    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 05 02:17:59.325122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318786    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318793    2555 flags.go:64] FLAG: --node-status-max-images="50"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318797    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318800    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318803    2555 flags.go:64] FLAG: --pod-cidr=""
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318806    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318812    2555 flags.go:64] FLAG: --pod-manifest-path=""
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318815    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318819    2555 flags.go:64] FLAG: --pods-per-core="0"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318823    2555 flags.go:64] FLAG: --port="10250"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318827    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318829    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-031f2ba2e2b656ab8"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318833    2555 flags.go:64] FLAG: --qos-reserved=""
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318836    2555 flags.go:64] FLAG: --read-only-port="10255"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318839    2555 flags.go:64] FLAG: --register-node="true"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318842    2555 flags.go:64] FLAG: --register-with-taints=""
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318846    2555 flags.go:64] FLAG: --registry-burst="10"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318849    2555 flags.go:64] FLAG: --registry-qps="5"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318852    2555 flags.go:64] FLAG: --reserved-cpus=""
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318856    2555 flags.go:64] FLAG: --reserved-memory=""
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318860    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318863    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318866    2555 flags.go:64] FLAG: --rotate-certificates="false"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318869    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318872    2555 flags.go:64] FLAG: --runonce="false"
Sep 05 02:17:59.325749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318875    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318878    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318880    2555 flags.go:64] FLAG: --seccomp-default="false"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318883    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318886    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318889    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318892    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318895    2555 flags.go:64] FLAG: --storage-driver-password="root"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318898    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318901    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318904    2555 flags.go:64] FLAG: --storage-driver-user="root"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318907    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318910    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318913    2555 flags.go:64] FLAG: --system-cgroups=""
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318916    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318921    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318926    2555 flags.go:64] FLAG: --tls-cert-file=""
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318929    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318933    2555 flags.go:64] FLAG: --tls-min-version=""
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318936    2555 flags.go:64] FLAG: --tls-private-key-file=""
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318939    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318942    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318944    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318950    2555 flags.go:64] FLAG: --v="2"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318954    2555 flags.go:64] FLAG: --version="false"
Sep 05 02:17:59.326379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318958    2555 flags.go:64] FLAG: --vmodule=""
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318963    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.318966    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319103    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319108    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319111    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319114    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319117    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319122    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319126    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319129    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319132    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319135    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319138    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319141    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319144    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319147    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319149    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319152    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319155    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:59.327044 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319158    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319160    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319163    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319167    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319170    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319172    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319175    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319177    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319180    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319182    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319185    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319188    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319191    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319195    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319197    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319200    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319202    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319205    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319207    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:59.327674 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319210    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319212    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319215    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319218    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319221    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319224    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319226    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319229    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319232    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319234    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319237    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319239    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319242    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319245    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319247    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319250    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319253    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319256    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319259    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:59.328436 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319261    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319264    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319266    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319269    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319271    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319274    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319277    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319281    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319284    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319286    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319289    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319291    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319295    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319298    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319301    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319304    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319306    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319309    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319311    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:59.328945 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319314    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319316    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319319    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319321    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319324    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319327    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319329    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319332    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319334    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319337    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319342    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319344    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319347    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319349    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319353    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319356    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319360    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319362    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319365    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319368    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:59.329405 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319373    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319375    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319378    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319380    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.319383    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.320165    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.329278    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.329295    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.329311    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.329317    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329377    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329382    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329385    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329389    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329392    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:59.329922 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329397    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329401    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329404    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329407    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329411    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329414    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329417    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329421    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329424    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329427    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329430    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329433    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329435    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329438    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329441    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329443    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329446    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329448    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329451    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329454    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:59.330309 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329458    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329461    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329464    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329467    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329470    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329472    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329475    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329477    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329480    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329482    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329484    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329487    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329490    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329492    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329495    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329497    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329500    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329502    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329505    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329507    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:59.330883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329510    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329512    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329515    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329518    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329520    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329523    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329526    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329528    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329547    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329552    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329556    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329561    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329568    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329573    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329576    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329579    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329582    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329585    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329587    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:59.331389 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329590    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329592    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329595    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329598    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329601    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329604    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329606    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329609    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329611    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329614    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329617    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329619    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329621    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329624    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329627    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329630    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329633    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329637    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329640    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:59.331871 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329643    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329646    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329649    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329651    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329654    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329656    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329660    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329662    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329665    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329667    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329670    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329673    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329675    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329678    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329680    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:59.332376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329682    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.329688    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329812    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329818    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329821    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329824    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329828    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329831    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329833    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329836    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329838    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329841    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329844    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329847    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329850    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:59.332751 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329852    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329855    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329857    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329860    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329862    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329865    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329867    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329870    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329872    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329875    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329878    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329880    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329883    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329886    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329888    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329891    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329893    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329896    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329899    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:59.333127 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329901    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329904    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329906    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329909    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329912    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329914    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329917    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329919    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329923    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329925    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329929    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329933    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329937    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329940    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329943    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329946    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329949    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329952    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329954    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:59.333602 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329957    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329960    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329962    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329965    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329968    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329970    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329973    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329975    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329978    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329982    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329985    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329988    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329991    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329993    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329996    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.329998    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330001    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330006    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330009    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330011    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:59.334069 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330014    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330017    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330019    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330022    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330024    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330026    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330029    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330033    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330036    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330038    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330041    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330043    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330046    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330049    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330051    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330054    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330057    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330061    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330064    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330067    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:59.334714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330069    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:59.335207 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330072    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:59.335207 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330074    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:59.335207 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330077    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:59.335207 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330079    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:59.335207 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330082    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:59.335207 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330084    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:59.335207 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:17:59.330087    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:59.335207 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.330092    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:59.335207 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.330234    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 05 02:17:59.335207 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.334504    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 05 02:17:59.335651 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.335639    2555 server.go:1019] "Starting client certificate rotation"
Sep 05 02:17:59.335763 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.335744    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:59.335793 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.335779    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:59.338464 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.338451    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:59.343199 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.343179    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:59.353818 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.353802    2555 log.go:25] "Validated CRI v1 runtime API"
Sep 05 02:17:59.357607 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.357592    2555 log.go:25] "Validated CRI v1 image API"
Sep 05 02:17:59.360140 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.360125    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 05 02:17:59.363671 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.363649    2555 fs.go:135] Filesystem UUIDs: map[5faf2052-64db-4a9d-9cad-7828a9a50a42:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d9d1b929-05e4-4104-9295-209299cdb55a:/dev/nvme0n1p4]
Sep 05 02:17:59.363742 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.363670    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 05 02:17:59.365088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.365073    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:17:59.369202 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.369092    2555 manager.go:216] Machine: {Timestamp:2026-09-05 02:17:59.367638888 +0000 UTC m=+0.376135029 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100209 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24572c58de3b780d33e0be615a8d1f SystemUUID:ec24572c-58de-3b78-0d33-e0be615a8d1f BootID:353994db-330c-4608-96d5-f021c425b682 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a4:85:b2:49:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a4:85:b2:49:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:09:3d:82:15:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 05 02:17:59.369202 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.369196    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 05 02:17:59.369311 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.369276    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 05 02:17:59.372201 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.372173    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 05 02:17:59.372717 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.372204    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-239.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 05 02:17:59.374150 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.374139    2555 topology_manager.go:138] "Creating topology manager with none policy"
Sep 05 02:17:59.374180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.374153    2555 container_manager_linux.go:309] "Creating device plugin manager"
Sep 05 02:17:59.374180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.374162    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:59.374895 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.374884    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:59.376823 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.376812    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 05 02:17:59.381554 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.381528    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:59.381689 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.381679    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 05 02:17:59.381722 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.381711    2555 kubelet.go:486] "Attempting to sync node with API server"
Sep 05 02:17:59.381752 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.381725    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 05 02:17:59.381752 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.381744    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 05 02:17:59.381807 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.381754    2555 kubelet.go:398] "Adding apiserver pod source"
Sep 05 02:17:59.381807 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.381763    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 05 02:17:59.382628 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.382606    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:59.382709 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.382637    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:59.385939 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.385923    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 05 02:17:59.388498 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.388483    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 05 02:17:59.388587 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.388513    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 05 02:17:59.388703 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.388692    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 05 02:17:59.388741 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.388708    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 05 02:17:59.388741 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.388715    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 05 02:17:59.388741 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.388720    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 05 02:17:59.388741 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.388728    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 05 02:17:59.388741 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.388733    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 05 02:17:59.389297 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.389289    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 05 02:17:59.389329 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.389297    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 05 02:17:59.389329 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.389304    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 05 02:17:59.389329 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.389309    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 05 02:17:59.389329 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.389318    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 05 02:17:59.389329 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.389325    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 05 02:17:59.390740 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.390729    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 05 02:17:59.390740 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.390739    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 05 02:17:59.394276 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.394251    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-239.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 05 02:17:59.394341 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.394253    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 05 02:17:59.394455 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.394442    2555 server.go:1268] "Started kubelet"
Sep 05 02:17:59.394582 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.394552    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 05 02:17:59.394620 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.394561    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 05 02:17:59.394667 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.394635    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 05 02:17:59.395241 ip-10-0-135-239 systemd[1]: Started Kubernetes Kubelet.
Sep 05 02:17:59.396003 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.395988    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 05 02:17:59.398664 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.398641    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 05 02:17:59.399449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.399437    2555 server.go:310] "Adding debug handlers to kubelet server"
Sep 05 02:17:59.401801 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.401784    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 05 02:17:59.401884 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.401799    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 05 02:17:59.401953 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.401919    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:17:59.402637 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402599    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 05 02:17:59.402731 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402650    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 05 02:17:59.402731 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402663    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 05 02:17:59.402802 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402731    2555 reconstruct.go:97] "Volume reconstruction finished"
Sep 05 02:17:59.402802 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402741    2555 reconciler.go:29] "Reconciler: start to sync state"
Sep 05 02:17:59.402885 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402849    2555 factory.go:153] Registering CRI-O factory
Sep 05 02:17:59.402885 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402866    2555 factory.go:223] Registration of the crio container factory successfully
Sep 05 02:17:59.402971 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402916    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 05 02:17:59.402971 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402924    2555 factory.go:55] Registering systemd factory
Sep 05 02:17:59.402971 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402932    2555 factory.go:223] Registration of the systemd container factory successfully
Sep 05 02:17:59.402971 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402946    2555 factory.go:103] Registering Raw factory
Sep 05 02:17:59.402971 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.402962    2555 manager.go:1196] Started watching for new ooms in manager
Sep 05 02:17:59.403459 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.403441    2555 manager.go:349] Starting recovery of all containers
Sep 05 02:17:59.404677 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.403440    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-239.ec2.internal.18d24b31d38fee76  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-239.ec2.internal,UID:ip-10-0-135-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-239.ec2.internal,},FirstTimestamp:2026-09-05 02:17:59.394414198 +0000 UTC m=+0.402910328,LastTimestamp:2026-09-05 02:17:59.394414198 +0000 UTC m=+0.402910328,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-239.ec2.internal,}"
Sep 05 02:17:59.404677 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.403489    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-239.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 05 02:17:59.408162 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.408129    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-239.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 05 02:17:59.408278 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.408241    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 05 02:17:59.413803 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.413780    2555 manager.go:354] Recovery completed
Sep 05 02:17:59.417881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.417865    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:59.419769 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.419751    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:59.419841 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.419782    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:59.419841 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.419793    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:59.420298 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.420284    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 05 02:17:59.420298 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.420298    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 05 02:17:59.420374 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.420304    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:59.420374 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.420313    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:59.420965 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.420951    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:59.421014 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.420976    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:59.421014 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.420988    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:59.422506 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.422492    2555 policy_none.go:49] "None policy: Start"
Sep 05 02:17:59.422585 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.422508    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 05 02:17:59.422585 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.422519    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 05 02:17:59.424045 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.421976    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-239.ec2.internal.18d24b31d512d06e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-239.ec2.internal,UID:ip-10-0-135-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-239.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-239.ec2.internal,},FirstTimestamp:2026-09-05 02:17:59.419768942 +0000 UTC m=+0.428265072,LastTimestamp:2026-09-05 02:17:59.419768942 +0000 UTC m=+0.428265072,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-239.ec2.internal,}"
Sep 05 02:17:59.424596 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.424578    2555 policy_none.go:47] "Start"
Sep 05 02:17:59.432615 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.432548    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-239.ec2.internal.18d24b31d5131900  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-239.ec2.internal,UID:ip-10-0-135-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-239.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-239.ec2.internal,},FirstTimestamp:2026-09-05 02:17:59.41978752 +0000 UTC m=+0.428283650,LastTimestamp:2026-09-05 02:17:59.41978752 +0000 UTC m=+0.428283650,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-239.ec2.internal,}"
Sep 05 02:17:59.440850 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.440772    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-239.ec2.internal.18d24b31d5133e81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-239.ec2.internal,UID:ip-10-0-135-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-239.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-239.ec2.internal,},FirstTimestamp:2026-09-05 02:17:59.419797121 +0000 UTC m=+0.428293251,LastTimestamp:2026-09-05 02:17:59.419797121 +0000 UTC m=+0.428293251,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-239.ec2.internal,}"
Sep 05 02:17:59.451798 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.451731    2555 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-239.ec2.internal.18d24b31d512d06e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-239.ec2.internal.18d24b31d512d06e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-239.ec2.internal,UID:ip-10-0-135-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-239.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-239.ec2.internal,},FirstTimestamp:2026-09-05 02:17:59.419768942 +0000 UTC m=+0.428265072,LastTimestamp:2026-09-05 02:17:59.420964429 +0000 UTC m=+0.429460559,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-239.ec2.internal,}"
Sep 05 02:17:59.454212 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.454199    2555 manager.go:337] "Starting Device Plugin manager"
Sep 05 02:17:59.454276 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.454247    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 05 02:17:59.454276 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.454261    2555 server.go:87] "Starting device plugin registration server"
Sep 05 02:17:59.454516 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.454502    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 05 02:17:59.454652 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.454516    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 05 02:17:59.454652 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.454645    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 05 02:17:59.454771 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.454730    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 05 02:17:59.454771 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.454740    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 05 02:17:59.455220 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.455198    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 05 02:17:59.455290 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.455248    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:17:59.461809 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.461735    2555 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-239.ec2.internal.18d24b31d5131900\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-239.ec2.internal.18d24b31d5131900  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-239.ec2.internal,UID:ip-10-0-135-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-239.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-239.ec2.internal,},FirstTimestamp:2026-09-05 02:17:59.41978752 +0000 UTC m=+0.428283650,LastTimestamp:2026-09-05 02:17:59.420980609 +0000 UTC m=+0.429476739,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-239.ec2.internal,}"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.463677    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zsq4n"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.471028    2555 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-239.ec2.internal.18d24b31d5133e81\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-239.ec2.internal.18d24b31d5133e81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-239.ec2.internal,UID:ip-10-0-135-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-239.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-239.ec2.internal,},FirstTimestamp:2026-09-05 02:17:59.419797121 +0000 UTC m=+0.428293251,LastTimestamp:2026-09-05 02:17:59.420993939 +0000 UTC m=+0.429490069,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-239.ec2.internal,}"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.472181    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zsq4n"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.536386    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.537590    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.537640    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.537668    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.537712    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.540166    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.555132    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.556049    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.556080    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.556095    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.556125    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.563759    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.563778    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-239.ec2.internal\": node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:17:59.591700 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.581003    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:17:59.638082 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.638032    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-239.ec2.internal"]
Sep 05 02:17:59.638244 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.638115    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:59.639638 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.639618    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:59.639728 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.639653    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:59.639728 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.639668    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:59.641905 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.641892    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:59.642010 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.641996    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.642055 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.642023    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:59.642884 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.642868    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:59.642884 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.642877    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:59.643035 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.642906    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:59.643035 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.642922    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:59.643035 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.642971    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:59.643035 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.642982    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:59.645182 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.645168    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.645232 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.645192    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:59.647007 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.646979    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:59.647077 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.647013    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:59.647077 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.647026    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:59.671332 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.671307    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-239.ec2.internal\" not found" node="ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.675686 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.675669    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-239.ec2.internal\" not found" node="ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.681274 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.681259    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:17:59.704320 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.704285    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ee890bde32bbb8ee31046089c37406d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-239.ec2.internal\" (UID: \"1ee890bde32bbb8ee31046089c37406d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.704320 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.704320    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0649b03eb74bf117dfbe86c89ad37396-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal\" (UID: \"0649b03eb74bf117dfbe86c89ad37396\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.704475 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.704338    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0649b03eb74bf117dfbe86c89ad37396-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal\" (UID: \"0649b03eb74bf117dfbe86c89ad37396\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.781694 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.781660    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:17:59.805056 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.805023    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0649b03eb74bf117dfbe86c89ad37396-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal\" (UID: \"0649b03eb74bf117dfbe86c89ad37396\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.805136 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.805066    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0649b03eb74bf117dfbe86c89ad37396-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal\" (UID: \"0649b03eb74bf117dfbe86c89ad37396\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.805136 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.805082    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ee890bde32bbb8ee31046089c37406d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-239.ec2.internal\" (UID: \"1ee890bde32bbb8ee31046089c37406d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.805136 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.805112    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0649b03eb74bf117dfbe86c89ad37396-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal\" (UID: \"0649b03eb74bf117dfbe86c89ad37396\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.805231 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.805145    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ee890bde32bbb8ee31046089c37406d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-239.ec2.internal\" (UID: \"1ee890bde32bbb8ee31046089c37406d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.805231 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.805148    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0649b03eb74bf117dfbe86c89ad37396-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal\" (UID: \"0649b03eb74bf117dfbe86c89ad37396\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.882355 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.882277    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:17:59.973936 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.973898    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.978768 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:17:59.978556    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-239.ec2.internal"
Sep 05 02:17:59.983089 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:17:59.983071    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:18:00.083677 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:00.083641    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:18:00.184264 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:00.184197    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:18:00.284805 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:00.284778    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:18:00.335718 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.335677    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 05 02:18:00.336154 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.335856    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:18:00.385842 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:00.385812    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:18:00.463429 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.463347    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:00.475905 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.475877    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 02:12:59 +0000 UTC" deadline="2028-02-16 03:16:29.563454893 +0000 UTC"
Sep 05 02:18:00.475970 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.475905    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12696h58m29.087552509s"
Sep 05 02:18:00.476104 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.476090    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:00.486205 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:00.486181    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:18:00.575919 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.575879    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:00.577303 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.577281    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:00.577421 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.577318    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:00.577421 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.577330    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:00.586609 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:00.586581    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:18:00.687909 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:00.687866    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:18:00.712565 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:00.712535    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee890bde32bbb8ee31046089c37406d.slice/crio-a0b2e57cb94b39d3da2e47b0f1b3bbe1f550b4725879622441a8d372534d2a4b WatchSource:0}: Error finding container a0b2e57cb94b39d3da2e47b0f1b3bbe1f550b4725879622441a8d372534d2a4b: Status 404 returned error can't find the container with id a0b2e57cb94b39d3da2e47b0f1b3bbe1f550b4725879622441a8d372534d2a4b
Sep 05 02:18:00.712830 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:00.712814    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0649b03eb74bf117dfbe86c89ad37396.slice/crio-5b66d5d868f757b116bf58c5e083a249a77962429664521812235f4d31628aa7 WatchSource:0}: Error finding container 5b66d5d868f757b116bf58c5e083a249a77962429664521812235f4d31628aa7: Status 404 returned error can't find the container with id 5b66d5d868f757b116bf58c5e083a249a77962429664521812235f4d31628aa7
Sep 05 02:18:00.717869 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.717852    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:18:00.788493 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:00.788438    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:18:00.889256 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:00.889207    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-239.ec2.internal\" not found"
Sep 05 02:18:00.975236 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:00.975139    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:01.002796 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.002769    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal"
Sep 05 02:18:01.013842 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.013820    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:18:01.014708 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.014694    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-239.ec2.internal"
Sep 05 02:18:01.023171 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.023150    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:18:01.384507 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.384428    2555 apiserver.go:52] "Watching apiserver"
Sep 05 02:18:01.392451 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.392423    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 05 02:18:01.393523 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.393492    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-96mr6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68","openshift-cluster-node-tuning-operator/tuned-vtds2","openshift-multus/multus-additional-cni-plugins-qb2xw","openshift-multus/multus-xbqkm","openshift-multus/network-metrics-daemon-85xds","openshift-network-diagnostics/network-check-target-8t685","openshift-ovn-kubernetes/ovnkube-node-2sg7d","kube-system/global-pull-secret-syncer-8dkfv","kube-system/kube-apiserver-proxy-ip-10-0-135-239.ec2.internal","openshift-dns/node-resolver-rfzkh","openshift-image-registry/node-ca-8zqc9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal","openshift-network-operator/iptables-alerter-bx78n"]
Sep 05 02:18:01.396170 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.396147    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:01.396278 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:01.396227    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:01.398275 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.398253    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rfzkh"
Sep 05 02:18:01.400891 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.400419    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.400989 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.400921    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tjnt6\""
Sep 05 02:18:01.401148 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.401130    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 05 02:18:01.401328 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.401311    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 05 02:18:01.402778 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.402624    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.402778 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.402715    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:01.402924 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.402836    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 05 02:18:01.403368 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.402715    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jmvnt\""
Sep 05 02:18:01.403449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.402835    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 05 02:18:01.404954 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.404775    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.405361 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.405341    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 05 02:18:01.405450 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.405368    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 05 02:18:01.405633 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.405612    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ctzzl\""
Sep 05 02:18:01.406667 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.406646    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 05 02:18:01.407114 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.406865    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 05 02:18:01.407114 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.406962    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 05 02:18:01.407114 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.407002    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 05 02:18:01.407267 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.407121    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:01.407267 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:01.407189    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:01.407364 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.407269    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c7jxq\""
Sep 05 02:18:01.409401 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.409382    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:01.409555 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:01.409469    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:01.411575 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.411557    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bx78n"
Sep 05 02:18:01.413914 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.413892    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 05 02:18:01.414106 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.414090    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 05 02:18:01.414311 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.414293    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sblq5\""
Sep 05 02:18:01.414726 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.414711    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.419167 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.419123    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.419858 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.419839    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 05 02:18:01.420137 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.420120    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 05 02:18:01.420239 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.420221    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:01.420525 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.420509    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 05 02:18:01.420663 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.420648    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c8jfl\""
Sep 05 02:18:01.420999 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.420973    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-sys\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.421086 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421064    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-lib-modules\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.421149 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421114    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-host\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.421149 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421143    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7ct7\" (UniqueName: \"kubernetes.io/projected/897e2168-985a-4ac7-95eb-9eb7cf09aa12-kube-api-access-g7ct7\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.421252 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421175    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-cnibin\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.421252 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421208    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44lhh\" (UniqueName: \"kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh\") pod \"network-check-target-8t685\" (UID: \"df75f0b8-d51e-4c43-a501-58dc006fef8e\") " pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:01.421252 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421242    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-socket-dir-parent\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.421808 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421787    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-daemon-config\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.421924 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421827    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlm7s\" (UniqueName: \"kubernetes.io/projected/57d95fd2-fff8-4df2-9ae3-89d120559276-kube-api-access-zlm7s\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.421924 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421858    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qsrf\" (UniqueName: \"kubernetes.io/projected/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-kube-api-access-5qsrf\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.421924 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421885    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-system-cni-dir\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.421924 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421913    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e1b36b-9b93-4340-a648-fafc7e0b6f90-tmp-dir\") pod \"node-resolver-rfzkh\" (UID: \"09e1b36b-9b93-4340-a648-fafc7e0b6f90\") " pod="openshift-dns/node-resolver-rfzkh"
Sep 05 02:18:01.422120 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421949    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzhll\" (UniqueName: \"kubernetes.io/projected/09e1b36b-9b93-4340-a648-fafc7e0b6f90-kube-api-access-gzhll\") pod \"node-resolver-rfzkh\" (UID: \"09e1b36b-9b93-4340-a648-fafc7e0b6f90\") " pod="openshift-dns/node-resolver-rfzkh"
Sep 05 02:18:01.422120 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.421979    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/897e2168-985a-4ac7-95eb-9eb7cf09aa12-tmp\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.422120 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422028    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.422120 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422062    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.422120 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422091    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 05 02:18:01.422352 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-systemd\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.422352 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422198    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-tuned\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.422352 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422241    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.422352 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422272    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-cni-dir\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.422352 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422310    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 05 02:18:01.422352 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422323    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-var-lib-cni-bin\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.422643 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422363    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-var-lib-kubelet\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.422643 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422415    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-kubernetes\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.422643 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422451    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-run-k8s-cni-cncf-io\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.422643 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422499    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 05 02:18:01.422643 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422504    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-sysctl-conf\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.422643 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422528    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vjtz8\""
Sep 05 02:18:01.422643 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422548    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57d95fd2-fff8-4df2-9ae3-89d120559276-cni-binary-copy\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.422643 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422577    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 05 02:18:01.422643 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422581    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-run-netns\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.422643 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422645    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-modprobe-d\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.423088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422675    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-sysctl-d\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.423088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-var-lib-kubelet\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.423088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422821    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:01.423088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422887    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-system-cni-dir\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.423088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422928    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.423088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422960    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-var-lib-cni-multus\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.423088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.422990    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-hostroot\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.423088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.423020    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-etc-kubernetes\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.423088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.423046    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-sysconfig\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.423088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.423079    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-run\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.423576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.423110    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-cnibin\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.423576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.423143    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-conf-dir\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.423576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.423204    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-run-multus-certs\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.423576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.423247    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09e1b36b-9b93-4340-a648-fafc7e0b6f90-hosts-file\") pod \"node-resolver-rfzkh\" (UID: \"09e1b36b-9b93-4340-a648-fafc7e0b6f90\") " pod="openshift-dns/node-resolver-rfzkh"
Sep 05 02:18:01.423576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.423554    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 05 02:18:01.423815 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.423631    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:18:01.423815 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.423557    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 05 02:18:01.425208 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.425043    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4g8p2\""
Sep 05 02:18:01.425440 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.425418    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8zqc9"
Sep 05 02:18:01.427359 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.427183    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 05 02:18:01.427359 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.427267    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 05 02:18:01.431884 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.431867    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 05 02:18:01.431973 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.431949    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 05 02:18:01.432118 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.432106    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-76mrd\""
Sep 05 02:18:01.432188 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.432151    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 05 02:18:01.444186 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.444163    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pb5gv"
Sep 05 02:18:01.452334 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.452313    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pb5gv"
Sep 05 02:18:01.504097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.504064    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 05 02:18:01.523770 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.523736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-run-netns\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.523928 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.523780    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-registration-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.523928 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.523810    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7bv2\" (UniqueName: \"kubernetes.io/projected/ef7a3697-f377-4d16-afe3-c8c4108fccff-kube-api-access-w7bv2\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.523928 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.523834    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:01.523928 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.523864    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-sys\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.523928 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.523904    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7b8c1413-db22-4560-9991-6260629f38be-iptables-alerter-script\") pod \"iptables-alerter-bx78n\" (UID: \"7b8c1413-db22-4560-9991-6260629f38be\") " pod="openshift-network-operator/iptables-alerter-bx78n"
Sep 05 02:18:01.524180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.523931    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-socket-dir-parent\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.524180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.523955    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-daemon-config\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.524180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.523981    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlm7s\" (UniqueName: \"kubernetes.io/projected/57d95fd2-fff8-4df2-9ae3-89d120559276-kube-api-access-zlm7s\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.524180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524026    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-sys\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.524180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524034    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-systemd-units\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.524492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524179    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-socket-dir-parent\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.524492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524220    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.524492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524249    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-ovnkube-script-lib\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.524492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524302    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06c658b0-5ad2-43c5-8057-633c5525d4ab-host\") pod \"node-ca-8zqc9\" (UID: \"06c658b0-5ad2-43c5-8057-633c5525d4ab\") " pod="openshift-image-registry/node-ca-8zqc9"
Sep 05 02:18:01.524492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524337    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/897e2168-985a-4ac7-95eb-9eb7cf09aa12-tmp\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.524492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524371    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.524847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524519    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84p95\" (UniqueName: \"kubernetes.io/projected/7b8c1413-db22-4560-9991-6260629f38be-kube-api-access-84p95\") pod \"iptables-alerter-bx78n\" (UID: \"7b8c1413-db22-4560-9991-6260629f38be\") " pod="openshift-network-operator/iptables-alerter-bx78n"
Sep 05 02:18:01.524847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524588    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-tuned\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.524847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524614    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.524847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-var-lib-cni-bin\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.524847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.524750    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 05 02:18:01.525205 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525180    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.525279 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525223    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.525279 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525233    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-ovn-node-metrics-cert\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.525279 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525262    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd509ebe-1474-4b42-88ae-96220ceecee8-dbus\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:01.525425 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525286    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-cni-netd\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.525425 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525332    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57d95fd2-fff8-4df2-9ae3-89d120559276-cni-binary-copy\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.525425 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-kubelet\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.525425 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-run-systemd\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.525425 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525400    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.525684 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525429    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-system-cni-dir\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.525684 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525458    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-hostroot\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.525684 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525495    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-etc-kubernetes\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.525684 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525518    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-etc-openvswitch\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.525684 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525556    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-run-openvswitch\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.525684 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525579    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-log-socket\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.525684 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525621    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-var-lib-cni-bin\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.525684 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.525647    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-hostroot\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.526129 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526099    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtmn8\" (UniqueName: \"kubernetes.io/projected/06c658b0-5ad2-43c5-8057-633c5525d4ab-kube-api-access-jtmn8\") pod \"node-ca-8zqc9\" (UID: \"06c658b0-5ad2-43c5-8057-633c5525d4ab\") " pod="openshift-image-registry/node-ca-8zqc9"
Sep 05 02:18:01.526129 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526110    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-daemon-config\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.526258 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526150    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-system-cni-dir\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.526258 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526221    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-sysconfig\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.526258 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526247    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-conf-dir\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.526393 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526281    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-run-multus-certs\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.526393 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526308    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clwks\" (UniqueName: \"kubernetes.io/projected/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-kube-api-access-clwks\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.526393 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526314    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-conf-dir\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.526393 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526333    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06c658b0-5ad2-43c5-8057-633c5525d4ab-serviceca\") pod \"node-ca-8zqc9\" (UID: \"06c658b0-5ad2-43c5-8057-633c5525d4ab\") " pod="openshift-image-registry/node-ca-8zqc9"
Sep 05 02:18:01.526393 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526372    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-etc-kubernetes\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.526393 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526376    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.526672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526408    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89c89ec9-f23a-4b39-83ed-d04f5c12362e-konnectivity-ca\") pod \"konnectivity-agent-96mr6\" (UID: \"89c89ec9-f23a-4b39-83ed-d04f5c12362e\") " pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:01.526672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526426    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-sysconfig\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.526672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526436    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09e1b36b-9b93-4340-a648-fafc7e0b6f90-hosts-file\") pod \"node-resolver-rfzkh\" (UID: \"09e1b36b-9b93-4340-a648-fafc7e0b6f90\") " pod="openshift-dns/node-resolver-rfzkh"
Sep 05 02:18:01.526672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526477    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-run-multus-certs\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.526672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526494    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09e1b36b-9b93-4340-a648-fafc7e0b6f90-hosts-file\") pod \"node-resolver-rfzkh\" (UID: \"09e1b36b-9b93-4340-a648-fafc7e0b6f90\") " pod="openshift-dns/node-resolver-rfzkh"
Sep 05 02:18:01.526672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526513    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-lib-modules\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.526672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526553    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-host\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.526672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526589    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-host\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.526672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526585    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7ct7\" (UniqueName: \"kubernetes.io/projected/897e2168-985a-4ac7-95eb-9eb7cf09aa12-kube-api-access-g7ct7\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.526672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526662    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-lib-modules\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526687    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-cnibin\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lhh\" (UniqueName: \"kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh\") pod \"network-check-target-8t685\" (UID: \"df75f0b8-d51e-4c43-a501-58dc006fef8e\") " pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-sys-fs\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526786    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m45fq\" (UniqueName: \"kubernetes.io/projected/2dd23bdc-d907-4b00-95c5-24828e92da98-kube-api-access-m45fq\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526806    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b8c1413-db22-4560-9991-6260629f38be-host-slash\") pod \"iptables-alerter-bx78n\" (UID: \"7b8c1413-db22-4560-9991-6260629f38be\") " pod="openshift-network-operator/iptables-alerter-bx78n"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526811    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-cnibin\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526829    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qsrf\" (UniqueName: \"kubernetes.io/projected/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-kube-api-access-5qsrf\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526860    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-system-cni-dir\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526887    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e1b36b-9b93-4340-a648-fafc7e0b6f90-tmp-dir\") pod \"node-resolver-rfzkh\" (UID: \"09e1b36b-9b93-4340-a648-fafc7e0b6f90\") " pod="openshift-dns/node-resolver-rfzkh"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526947    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzhll\" (UniqueName: \"kubernetes.io/projected/09e1b36b-9b93-4340-a648-fafc7e0b6f90-kube-api-access-gzhll\") pod \"node-resolver-rfzkh\" (UID: \"09e1b36b-9b93-4340-a648-fafc7e0b6f90\") " pod="openshift-dns/node-resolver-rfzkh"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526970    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-slash\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.526992    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-var-lib-openvswitch\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-env-overrides\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527042    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527069    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-systemd\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.527124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527102    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-cni-dir\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.527677 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527224    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-multus-cni-dir\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.527677 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527389    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-system-cni-dir\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.527677 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e1b36b-9b93-4340-a648-fafc7e0b6f90-tmp-dir\") pod \"node-resolver-rfzkh\" (UID: \"09e1b36b-9b93-4340-a648-fafc7e0b6f90\") " pod="openshift-dns/node-resolver-rfzkh"
Sep 05 02:18:01.527677 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527593    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-var-lib-kubelet\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.527677 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527625    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd509ebe-1474-4b42-88ae-96220ceecee8-kubelet-config\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:01.527677 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-kubernetes\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.527847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527722    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-run-k8s-cni-cncf-io\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.527847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527748    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-cni-bin\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.527847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527774    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89c89ec9-f23a-4b39-83ed-d04f5c12362e-agent-certs\") pod \"konnectivity-agent-96mr6\" (UID: \"89c89ec9-f23a-4b39-83ed-d04f5c12362e\") " pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:01.527847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527774    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.527847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527802    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-sysctl-conf\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.527847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527832    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-run-netns\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.527847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527836    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-systemd\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.528042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527896    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-node-log\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.528042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527922    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:01.528042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527948    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.528042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.527976    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-socket-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.528042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528000    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57d95fd2-fff8-4df2-9ae3-89d120559276-cni-binary-copy\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.528042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528001    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-run-netns\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.528042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528022    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-modprobe-d\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.528042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528041    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-kubernetes\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528050    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-sysctl-d\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528071    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-run-k8s-cni-cncf-io\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528103    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-var-lib-kubelet\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528107    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-var-lib-kubelet\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528134    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528169    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-sysctl-conf\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528180    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-var-lib-cni-multus\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528187    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-modprobe-d\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528214    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-run-ovn\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-host-var-lib-cni-multus\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528258    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-ovnkube-config\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528316    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-var-lib-kubelet\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528342    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-device-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528368    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-run\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528377    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-sysctl-d\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.528411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528393    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-cnibin\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.529194 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57d95fd2-fff8-4df2-9ae3-89d120559276-cnibin\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.529194 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528492    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/897e2168-985a-4ac7-95eb-9eb7cf09aa12-run\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.529194 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.528618    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.531659 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.531637    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/897e2168-985a-4ac7-95eb-9eb7cf09aa12-etc-tuned\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.532468 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.532445    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/897e2168-985a-4ac7-95eb-9eb7cf09aa12-tmp\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.533793 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:01.533487    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:01.533793 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:01.533513    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:01.533793 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:01.533526    2555 projected.go:196] Error preparing data for projected volume kube-api-access-44lhh for pod openshift-network-diagnostics/network-check-target-8t685: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:01.533793 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:01.533640    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh podName:df75f0b8-d51e-4c43-a501-58dc006fef8e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:02.033618429 +0000 UTC m=+3.042114558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-44lhh" (UniqueName: "kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh") pod "network-check-target-8t685" (UID: "df75f0b8-d51e-4c43-a501-58dc006fef8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:01.536945 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.536911    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7ct7\" (UniqueName: \"kubernetes.io/projected/897e2168-985a-4ac7-95eb-9eb7cf09aa12-kube-api-access-g7ct7\") pod \"tuned-vtds2\" (UID: \"897e2168-985a-4ac7-95eb-9eb7cf09aa12\") " pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.537621 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.537577    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlm7s\" (UniqueName: \"kubernetes.io/projected/57d95fd2-fff8-4df2-9ae3-89d120559276-kube-api-access-zlm7s\") pod \"multus-xbqkm\" (UID: \"57d95fd2-fff8-4df2-9ae3-89d120559276\") " pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.538955 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.538599    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzhll\" (UniqueName: \"kubernetes.io/projected/09e1b36b-9b93-4340-a648-fafc7e0b6f90-kube-api-access-gzhll\") pod \"node-resolver-rfzkh\" (UID: \"09e1b36b-9b93-4340-a648-fafc7e0b6f90\") " pod="openshift-dns/node-resolver-rfzkh"
Sep 05 02:18:01.540248 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.540205    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qsrf\" (UniqueName: \"kubernetes.io/projected/5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae-kube-api-access-5qsrf\") pod \"multus-additional-cni-plugins-qb2xw\" (UID: \"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae\") " pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.546981 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.546906    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal" event={"ID":"0649b03eb74bf117dfbe86c89ad37396","Type":"ContainerStarted","Data":"5b66d5d868f757b116bf58c5e083a249a77962429664521812235f4d31628aa7"}
Sep 05 02:18:01.548487 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.548466    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-239.ec2.internal" event={"ID":"1ee890bde32bbb8ee31046089c37406d","Type":"ContainerStarted","Data":"a0b2e57cb94b39d3da2e47b0f1b3bbe1f550b4725879622441a8d372534d2a4b"}
Sep 05 02:18:01.628761 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628728    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-kubelet\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.628932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-run-systemd\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.628932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628793    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.628932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628817    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-etc-openvswitch\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.628932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628838    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-run-openvswitch\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.628932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628845    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-kubelet\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.628932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628849    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-run-systemd\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.628932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628856    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-log-socket\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.628932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628891    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-log-socket\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.628932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628903    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-run-ovn-kubernetes\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.628932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628916    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtmn8\" (UniqueName: \"kubernetes.io/projected/06c658b0-5ad2-43c5-8057-633c5525d4ab-kube-api-access-jtmn8\") pod \"node-ca-8zqc9\" (UID: \"06c658b0-5ad2-43c5-8057-633c5525d4ab\") " pod="openshift-image-registry/node-ca-8zqc9"
Sep 05 02:18:01.628932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628883    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-etc-openvswitch\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.628937    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-run-openvswitch\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629022    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clwks\" (UniqueName: \"kubernetes.io/projected/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-kube-api-access-clwks\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629049    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06c658b0-5ad2-43c5-8057-633c5525d4ab-serviceca\") pod \"node-ca-8zqc9\" (UID: \"06c658b0-5ad2-43c5-8057-633c5525d4ab\") " pod="openshift-image-registry/node-ca-8zqc9"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629075    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629100    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89c89ec9-f23a-4b39-83ed-d04f5c12362e-konnectivity-ca\") pod \"konnectivity-agent-96mr6\" (UID: \"89c89ec9-f23a-4b39-83ed-d04f5c12362e\") " pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629142    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-sys-fs\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m45fq\" (UniqueName: \"kubernetes.io/projected/2dd23bdc-d907-4b00-95c5-24828e92da98-kube-api-access-m45fq\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629190    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b8c1413-db22-4560-9991-6260629f38be-host-slash\") pod \"iptables-alerter-bx78n\" (UID: \"7b8c1413-db22-4560-9991-6260629f38be\") " pod="openshift-network-operator/iptables-alerter-bx78n"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629220    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-slash\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629242    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-var-lib-openvswitch\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629273    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-env-overrides\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629295    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-slash\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd509ebe-1474-4b42-88ae-96220ceecee8-kubelet-config\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629334    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-cni-bin\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629358    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89c89ec9-f23a-4b39-83ed-d04f5c12362e-agent-certs\") pod \"konnectivity-agent-96mr6\" (UID: \"89c89ec9-f23a-4b39-83ed-d04f5c12362e\") " pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:01.629460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-node-log\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629405    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629428    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-socket-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629477    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-run-ovn\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629503    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-ovnkube-config\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629505    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b8c1413-db22-4560-9991-6260629f38be-host-slash\") pod \"iptables-alerter-bx78n\" (UID: \"7b8c1413-db22-4560-9991-6260629f38be\") " pod="openshift-network-operator/iptables-alerter-bx78n"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629546    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06c658b0-5ad2-43c5-8057-633c5525d4ab-serviceca\") pod \"node-ca-8zqc9\" (UID: \"06c658b0-5ad2-43c5-8057-633c5525d4ab\") " pod="openshift-image-registry/node-ca-8zqc9"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629562    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-device-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629603    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-run-netns\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629612    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-node-log\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629618    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-run-ovn\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629628    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-registration-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629659    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7bv2\" (UniqueName: \"kubernetes.io/projected/ef7a3697-f377-4d16-afe3-c8c4108fccff-kube-api-access-w7bv2\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:01.629679    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629684    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629713    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7b8c1413-db22-4560-9991-6260629f38be-iptables-alerter-script\") pod \"iptables-alerter-bx78n\" (UID: \"7b8c1413-db22-4560-9991-6260629f38be\") " pod="openshift-network-operator/iptables-alerter-bx78n"
Sep 05 02:18:01.630240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629718    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-cni-bin\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:01.629736    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret podName:bd509ebe-1474-4b42-88ae-96220ceecee8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:02.129719594 +0000 UTC m=+3.138215715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret") pod "global-pull-secret-syncer-8dkfv" (UID: "bd509ebe-1474-4b42-88ae-96220ceecee8") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629680    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd509ebe-1474-4b42-88ae-96220ceecee8-kubelet-config\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629778    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-registration-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629786    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629824    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-device-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629849    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-run-netns\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629881    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89c89ec9-f23a-4b39-83ed-d04f5c12362e-konnectivity-ca\") pod \"konnectivity-agent-96mr6\" (UID: \"89c89ec9-f23a-4b39-83ed-d04f5c12362e\") " pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629728    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-socket-dir\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:01.629937    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef7a3697-f377-4d16-afe3-c8c4108fccff-sys-fs\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629967    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-systemd-units\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:01.629999    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs podName:2dd23bdc-d907-4b00-95c5-24828e92da98 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:02.129982737 +0000 UTC m=+3.138478855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs") pod "network-metrics-daemon-85xds" (UID: "2dd23bdc-d907-4b00-95c5-24828e92da98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.629939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-var-lib-openvswitch\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630022    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630024    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-systemd-units\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630051    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-env-overrides\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630063    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630058    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-ovnkube-script-lib\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630107    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06c658b0-5ad2-43c5-8057-633c5525d4ab-host\") pod \"node-ca-8zqc9\" (UID: \"06c658b0-5ad2-43c5-8057-633c5525d4ab\") " pod="openshift-image-registry/node-ca-8zqc9"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630144    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84p95\" (UniqueName: \"kubernetes.io/projected/7b8c1413-db22-4560-9991-6260629f38be-kube-api-access-84p95\") pod \"iptables-alerter-bx78n\" (UID: \"7b8c1413-db22-4560-9991-6260629f38be\") " pod="openshift-network-operator/iptables-alerter-bx78n"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630174    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-ovn-node-metrics-cert\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630199    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd509ebe-1474-4b42-88ae-96220ceecee8-dbus\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630211    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7b8c1413-db22-4560-9991-6260629f38be-iptables-alerter-script\") pod \"iptables-alerter-bx78n\" (UID: \"7b8c1413-db22-4560-9991-6260629f38be\") " pod="openshift-network-operator/iptables-alerter-bx78n"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630222    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-cni-netd\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06c658b0-5ad2-43c5-8057-633c5525d4ab-host\") pod \"node-ca-8zqc9\" (UID: \"06c658b0-5ad2-43c5-8057-633c5525d4ab\") " pod="openshift-image-registry/node-ca-8zqc9"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630481    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd509ebe-1474-4b42-88ae-96220ceecee8-dbus\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630484    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-host-cni-netd\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630579    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-ovnkube-script-lib\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.631856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.630687    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-ovnkube-config\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.632896 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.632792    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-ovn-node-metrics-cert\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.632896 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.632844    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89c89ec9-f23a-4b39-83ed-d04f5c12362e-agent-certs\") pod \"konnectivity-agent-96mr6\" (UID: \"89c89ec9-f23a-4b39-83ed-d04f5c12362e\") " pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:01.641829 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.641762    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtmn8\" (UniqueName: \"kubernetes.io/projected/06c658b0-5ad2-43c5-8057-633c5525d4ab-kube-api-access-jtmn8\") pod \"node-ca-8zqc9\" (UID: \"06c658b0-5ad2-43c5-8057-633c5525d4ab\") " pod="openshift-image-registry/node-ca-8zqc9"
Sep 05 02:18:01.643870 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.643837    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clwks\" (UniqueName: \"kubernetes.io/projected/d6059461-d098-4d3f-8ae4-11d5a1b09ff5-kube-api-access-clwks\") pod \"ovnkube-node-2sg7d\" (UID: \"d6059461-d098-4d3f-8ae4-11d5a1b09ff5\") " pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.644233 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.644205    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m45fq\" (UniqueName: \"kubernetes.io/projected/2dd23bdc-d907-4b00-95c5-24828e92da98-kube-api-access-m45fq\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:01.645621 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.645552    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7bv2\" (UniqueName: \"kubernetes.io/projected/ef7a3697-f377-4d16-afe3-c8c4108fccff-kube-api-access-w7bv2\") pod \"aws-ebs-csi-driver-node-hkx68\" (UID: \"ef7a3697-f377-4d16-afe3-c8c4108fccff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.645909 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.645890    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84p95\" (UniqueName: \"kubernetes.io/projected/7b8c1413-db22-4560-9991-6260629f38be-kube-api-access-84p95\") pod \"iptables-alerter-bx78n\" (UID: \"7b8c1413-db22-4560-9991-6260629f38be\") " pod="openshift-network-operator/iptables-alerter-bx78n"
Sep 05 02:18:01.712184 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.712146    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rfzkh"
Sep 05 02:18:01.723947 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.723917    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vtds2"
Sep 05 02:18:01.732762 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.732741    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qb2xw"
Sep 05 02:18:01.738380 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.738360    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:01.748018 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.747996    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xbqkm"
Sep 05 02:18:01.756608 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.756587    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bx78n"
Sep 05 02:18:01.764286 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.764260    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68"
Sep 05 02:18:01.772011 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.771995    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:01.777569 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.777552    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8zqc9"
Sep 05 02:18:01.826314 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:01.826289    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:02.133081 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:02.132990    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:02.133081 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:02.133044    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:02.133302 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:02.133099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lhh\" (UniqueName: \"kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh\") pod \"network-check-target-8t685\" (UID: \"df75f0b8-d51e-4c43-a501-58dc006fef8e\") " pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:02.133302 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:02.133167    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:02.133302 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:02.133242    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret podName:bd509ebe-1474-4b42-88ae-96220ceecee8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:03.133221284 +0000 UTC m=+4.141717424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret") pod "global-pull-secret-syncer-8dkfv" (UID: "bd509ebe-1474-4b42-88ae-96220ceecee8") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:02.133302 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:02.133243    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:02.133498 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:02.133308    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs podName:2dd23bdc-d907-4b00-95c5-24828e92da98 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:03.133288971 +0000 UTC m=+4.141785103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs") pod "network-metrics-daemon-85xds" (UID: "2dd23bdc-d907-4b00-95c5-24828e92da98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:02.133498 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:02.133318    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:02.133498 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:02.133336    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:02.133498 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:02.133349    2555 projected.go:196] Error preparing data for projected volume kube-api-access-44lhh for pod openshift-network-diagnostics/network-check-target-8t685: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:02.133498 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:02.133395    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh podName:df75f0b8-d51e-4c43-a501-58dc006fef8e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:03.133378535 +0000 UTC m=+4.141874655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-44lhh" (UniqueName: "kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh") pod "network-check-target-8t685" (UID: "df75f0b8-d51e-4c43-a501-58dc006fef8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:02.455490 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:02.455395    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:13:01 +0000 UTC" deadline="2028-03-23 19:32:34.423508027 +0000 UTC"
Sep 05 02:18:02.455490 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:02.455430    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13577h14m31.968080973s"
Sep 05 02:18:02.583933 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:02.583908    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e1b36b_9b93_4340_a648_fafc7e0b6f90.slice/crio-362c1a9967358addac669167f3652a207779e398f6415a2592904ee14b33ad54 WatchSource:0}: Error finding container 362c1a9967358addac669167f3652a207779e398f6415a2592904ee14b33ad54: Status 404 returned error can't find the container with id 362c1a9967358addac669167f3652a207779e398f6415a2592904ee14b33ad54
Sep 05 02:18:02.584983 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:02.584956    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c658b0_5ad2_43c5_8057_633c5525d4ab.slice/crio-83d3a905d0857e4c574eec6d16d497825ea3a69279cc1effad66f8513954bcee WatchSource:0}: Error finding container 83d3a905d0857e4c574eec6d16d497825ea3a69279cc1effad66f8513954bcee: Status 404 returned error can't find the container with id 83d3a905d0857e4c574eec6d16d497825ea3a69279cc1effad66f8513954bcee
Sep 05 02:18:02.587848 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:02.587797    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5709cf_8c39_4bd7_9f0a_0ce5afed64ae.slice/crio-09822ac9f28ba9c09eecfe447fa328f5050848073de9b11511dccbf94b2307e7 WatchSource:0}: Error finding container 09822ac9f28ba9c09eecfe447fa328f5050848073de9b11511dccbf94b2307e7: Status 404 returned error can't find the container with id 09822ac9f28ba9c09eecfe447fa328f5050848073de9b11511dccbf94b2307e7
Sep 05 02:18:02.588883 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:02.588862    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7a3697_f377_4d16_afe3_c8c4108fccff.slice/crio-5b9fcd311776abffaaca059ec6d900c75468cb38499f858207c8352b1c86e50e WatchSource:0}: Error finding container 5b9fcd311776abffaaca059ec6d900c75468cb38499f858207c8352b1c86e50e: Status 404 returned error can't find the container with id 5b9fcd311776abffaaca059ec6d900c75468cb38499f858207c8352b1c86e50e
Sep 05 02:18:02.610030 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:02.609941    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8c1413_db22_4560_9991_6260629f38be.slice/crio-8aa4a063dc00985ec4b5ce3896aeecd3d6f3d9b882bf8149665f9d6ec6844ddb WatchSource:0}: Error finding container 8aa4a063dc00985ec4b5ce3896aeecd3d6f3d9b882bf8149665f9d6ec6844ddb: Status 404 returned error can't find the container with id 8aa4a063dc00985ec4b5ce3896aeecd3d6f3d9b882bf8149665f9d6ec6844ddb
Sep 05 02:18:02.610861 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:02.610827    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897e2168_985a_4ac7_95eb_9eb7cf09aa12.slice/crio-dfdb3b8057c1ae839629cfb5649b94f1cb4434ad49a54d2abdbe1fd7869a67ac WatchSource:0}: Error finding container dfdb3b8057c1ae839629cfb5649b94f1cb4434ad49a54d2abdbe1fd7869a67ac: Status 404 returned error can't find the container with id dfdb3b8057c1ae839629cfb5649b94f1cb4434ad49a54d2abdbe1fd7869a67ac
Sep 05 02:18:02.611852 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:02.611816    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6059461_d098_4d3f_8ae4_11d5a1b09ff5.slice/crio-20a743d04ad08c70a08237ec956318566b7493ddd4873afe1e87a10c7a3320f9 WatchSource:0}: Error finding container 20a743d04ad08c70a08237ec956318566b7493ddd4873afe1e87a10c7a3320f9: Status 404 returned error can't find the container with id 20a743d04ad08c70a08237ec956318566b7493ddd4873afe1e87a10c7a3320f9
Sep 05 02:18:02.612666 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:02.612635    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c89ec9_f23a_4b39_83ed_d04f5c12362e.slice/crio-e9496e0bbbffa49cc59e769097bb1673d866288bba0d0311b07e47ef34b9af10 WatchSource:0}: Error finding container e9496e0bbbffa49cc59e769097bb1673d866288bba0d0311b07e47ef34b9af10: Status 404 returned error can't find the container with id e9496e0bbbffa49cc59e769097bb1673d866288bba0d0311b07e47ef34b9af10
Sep 05 02:18:02.613714 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:02.613639    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d95fd2_fff8_4df2_9ae3_89d120559276.slice/crio-0e0f0071e64678789dcde5215b83069f6bdcb79552979a5355810f05c121fbaf WatchSource:0}: Error finding container 0e0f0071e64678789dcde5215b83069f6bdcb79552979a5355810f05c121fbaf: Status 404 returned error can't find the container with id 0e0f0071e64678789dcde5215b83069f6bdcb79552979a5355810f05c121fbaf
Sep 05 02:18:03.140388 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.139311    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lhh\" (UniqueName: \"kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh\") pod \"network-check-target-8t685\" (UID: \"df75f0b8-d51e-4c43-a501-58dc006fef8e\") " pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:03.140388 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.139376    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:03.140388 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.139413    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:03.140388 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:03.139583    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:03.140388 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:03.139647    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs podName:2dd23bdc-d907-4b00-95c5-24828e92da98 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:05.13962767 +0000 UTC m=+6.148123801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs") pod "network-metrics-daemon-85xds" (UID: "2dd23bdc-d907-4b00-95c5-24828e92da98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:03.140388 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:03.140068    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:03.140388 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:03.140087    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:03.140388 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:03.140100    2555 projected.go:196] Error preparing data for projected volume kube-api-access-44lhh for pod openshift-network-diagnostics/network-check-target-8t685: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:03.140388 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:03.140149    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh podName:df75f0b8-d51e-4c43-a501-58dc006fef8e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:05.140133973 +0000 UTC m=+6.148630096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-44lhh" (UniqueName: "kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh") pod "network-check-target-8t685" (UID: "df75f0b8-d51e-4c43-a501-58dc006fef8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:03.140388 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:03.140214    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:03.140388 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:03.140251    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret podName:bd509ebe-1474-4b42-88ae-96220ceecee8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:05.140238094 +0000 UTC m=+6.148734219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret") pod "global-pull-secret-syncer-8dkfv" (UID: "bd509ebe-1474-4b42-88ae-96220ceecee8") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:03.455873 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.455762    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:13:01 +0000 UTC" deadline="2028-06-07 18:10:06.39702559 +0000 UTC"
Sep 05 02:18:03.455873 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.455820    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15399h52m2.941209387s"
Sep 05 02:18:03.538680 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.538650    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:03.538680 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.538668    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:03.538886 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:03.538779    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:03.538941 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:03.538874    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:03.538941 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.538936    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:03.539068 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:03.539007    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:03.560888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.560805    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-96mr6" event={"ID":"89c89ec9-f23a-4b39-83ed-d04f5c12362e","Type":"ContainerStarted","Data":"e9496e0bbbffa49cc59e769097bb1673d866288bba0d0311b07e47ef34b9af10"}
Sep 05 02:18:03.568227 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.568188    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d" event={"ID":"d6059461-d098-4d3f-8ae4-11d5a1b09ff5","Type":"ContainerStarted","Data":"20a743d04ad08c70a08237ec956318566b7493ddd4873afe1e87a10c7a3320f9"}
Sep 05 02:18:03.574384 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.574301    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vtds2" event={"ID":"897e2168-985a-4ac7-95eb-9eb7cf09aa12","Type":"ContainerStarted","Data":"dfdb3b8057c1ae839629cfb5649b94f1cb4434ad49a54d2abdbe1fd7869a67ac"}
Sep 05 02:18:03.583492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.583459    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bx78n" event={"ID":"7b8c1413-db22-4560-9991-6260629f38be","Type":"ContainerStarted","Data":"8aa4a063dc00985ec4b5ce3896aeecd3d6f3d9b882bf8149665f9d6ec6844ddb"}
Sep 05 02:18:03.586626 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.586557    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8zqc9" event={"ID":"06c658b0-5ad2-43c5-8057-633c5525d4ab","Type":"ContainerStarted","Data":"83d3a905d0857e4c574eec6d16d497825ea3a69279cc1effad66f8513954bcee"}
Sep 05 02:18:03.597584 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.597195    2555 generic.go:356] "Generic (PLEG): container finished" podID="0649b03eb74bf117dfbe86c89ad37396" containerID="94f4fe22801744430665e352042d5e829040ae3c6efe3a99d00b0408c30e2c3d" exitCode=0
Sep 05 02:18:03.597584 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.597254    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal" event={"ID":"0649b03eb74bf117dfbe86c89ad37396","Type":"ContainerDied","Data":"94f4fe22801744430665e352042d5e829040ae3c6efe3a99d00b0408c30e2c3d"}
Sep 05 02:18:03.611118 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.611090    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-239.ec2.internal" event={"ID":"1ee890bde32bbb8ee31046089c37406d","Type":"ContainerStarted","Data":"67463e893e6ff9bb40912210acf35cfe57271f5f61f3bf62cb3a60e2183c7773"}
Sep 05 02:18:03.625140 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.625049    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68" event={"ID":"ef7a3697-f377-4d16-afe3-c8c4108fccff","Type":"ContainerStarted","Data":"5b9fcd311776abffaaca059ec6d900c75468cb38499f858207c8352b1c86e50e"}
Sep 05 02:18:03.630250 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.630122    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-239.ec2.internal" podStartSLOduration=2.630104762 podStartE2EDuration="2.630104762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:03.629676951 +0000 UTC m=+4.638173118" watchObservedRunningTime="2026-09-05 02:18:03.630104762 +0000 UTC m=+4.638600907"
Sep 05 02:18:03.639066 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.639043    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb2xw" event={"ID":"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae","Type":"ContainerStarted","Data":"09822ac9f28ba9c09eecfe447fa328f5050848073de9b11511dccbf94b2307e7"}
Sep 05 02:18:03.650566 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.650505    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rfzkh" event={"ID":"09e1b36b-9b93-4340-a648-fafc7e0b6f90","Type":"ContainerStarted","Data":"362c1a9967358addac669167f3652a207779e398f6415a2592904ee14b33ad54"}
Sep 05 02:18:03.669566 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:03.669517    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xbqkm" event={"ID":"57d95fd2-fff8-4df2-9ae3-89d120559276","Type":"ContainerStarted","Data":"0e0f0071e64678789dcde5215b83069f6bdcb79552979a5355810f05c121fbaf"}
Sep 05 02:18:04.688679 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:04.688598    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal" event={"ID":"0649b03eb74bf117dfbe86c89ad37396","Type":"ContainerStarted","Data":"67a5836dc2d21be15d78dfe522fc070d74ca1f41c341635940ffab30927adf56"}
Sep 05 02:18:05.155738 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:05.155654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:05.155888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:05.155740    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lhh\" (UniqueName: \"kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh\") pod \"network-check-target-8t685\" (UID: \"df75f0b8-d51e-4c43-a501-58dc006fef8e\") " pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:05.155888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:05.155786    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:05.155888 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:05.155829    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:05.156050 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:05.155891    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:05.156050 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:05.155926    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs podName:2dd23bdc-d907-4b00-95c5-24828e92da98 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:09.155890472 +0000 UTC m=+10.164386608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs") pod "network-metrics-daemon-85xds" (UID: "2dd23bdc-d907-4b00-95c5-24828e92da98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:05.156050 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:05.155949    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret podName:bd509ebe-1474-4b42-88ae-96220ceecee8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:09.155939285 +0000 UTC m=+10.164435408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret") pod "global-pull-secret-syncer-8dkfv" (UID: "bd509ebe-1474-4b42-88ae-96220ceecee8") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:05.156050 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:05.156002    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:05.156050 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:05.156019    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:05.156050 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:05.156032    2555 projected.go:196] Error preparing data for projected volume kube-api-access-44lhh for pod openshift-network-diagnostics/network-check-target-8t685: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:05.156320 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:05.156071    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh podName:df75f0b8-d51e-4c43-a501-58dc006fef8e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:09.156055981 +0000 UTC m=+10.164552104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-44lhh" (UniqueName: "kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh") pod "network-check-target-8t685" (UID: "df75f0b8-d51e-4c43-a501-58dc006fef8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:05.538894 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:05.538795    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:05.539046 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:05.538947    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:05.539335 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:05.539314    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:05.539439 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:05.539409    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:05.539592 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:05.539549    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:05.539662 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:05.539641    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:07.538615 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:07.538497    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:07.539096 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:07.538649    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:07.539177 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:07.538497    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:07.539234 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:07.539181    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:07.539314 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:07.539290    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:07.539440 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:07.539407    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:09.190295 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:09.190249    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:09.190768 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:09.190320    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:09.190768 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:09.190382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lhh\" (UniqueName: \"kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh\") pod \"network-check-target-8t685\" (UID: \"df75f0b8-d51e-4c43-a501-58dc006fef8e\") " pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:09.190768 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:09.190489    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:09.190768 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:09.190509    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:09.190768 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:09.190524    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:09.190768 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:09.190551    2555 projected.go:196] Error preparing data for projected volume kube-api-access-44lhh for pod openshift-network-diagnostics/network-check-target-8t685: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:09.190768 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:09.190572    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret podName:bd509ebe-1474-4b42-88ae-96220ceecee8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:17.190553208 +0000 UTC m=+18.199049336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret") pod "global-pull-secret-syncer-8dkfv" (UID: "bd509ebe-1474-4b42-88ae-96220ceecee8") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:09.190768 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:09.190490    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:09.190768 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:09.190590    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh podName:df75f0b8-d51e-4c43-a501-58dc006fef8e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:17.190578807 +0000 UTC m=+18.199074926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-44lhh" (UniqueName: "kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh") pod "network-check-target-8t685" (UID: "df75f0b8-d51e-4c43-a501-58dc006fef8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:09.190768 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:09.190662    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs podName:2dd23bdc-d907-4b00-95c5-24828e92da98 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:17.190641724 +0000 UTC m=+18.199137848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs") pod "network-metrics-daemon-85xds" (UID: "2dd23bdc-d907-4b00-95c5-24828e92da98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:09.540064 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:09.539975    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:09.540064 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:09.539975    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:09.541160 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:09.541126    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:09.541287 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:09.541229    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:09.541349 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:09.541329    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:09.541460 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:09.541436    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:11.539066 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:11.539008    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:11.539066 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:11.539041    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:11.539557 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:11.539156    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:11.539557 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:11.539238    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:11.539557 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:11.539311    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:11.539557 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:11.539386    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:13.538428 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:13.538395    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:13.538948 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:13.538443    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:13.538948 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:13.538552    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:13.538948 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:13.538648    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:13.538948 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:13.538691    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:13.538948 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:13.538755    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:15.538709 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:15.538664    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:15.539172 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:15.538664    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:15.539172 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:15.538801    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:15.539172 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:15.538664    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:15.539172 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:15.538885    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:15.539172 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:15.539015    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:17.255760 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:17.255709    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lhh\" (UniqueName: \"kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh\") pod \"network-check-target-8t685\" (UID: \"df75f0b8-d51e-4c43-a501-58dc006fef8e\") " pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:17.256269 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:17.255773    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:17.256269 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:17.255816    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:17.256269 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:17.255881    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:17.256269 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:17.255893    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:17.256269 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:17.255914    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:17.256269 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:17.255921    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:17.256269 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:17.255927    2555 projected.go:196] Error preparing data for projected volume kube-api-access-44lhh for pod openshift-network-diagnostics/network-check-target-8t685: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:17.256269 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:17.255954    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret podName:bd509ebe-1474-4b42-88ae-96220ceecee8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:33.255934464 +0000 UTC m=+34.264430601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret") pod "global-pull-secret-syncer-8dkfv" (UID: "bd509ebe-1474-4b42-88ae-96220ceecee8") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:17.256269 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:17.255972    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh podName:df75f0b8-d51e-4c43-a501-58dc006fef8e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:33.255962347 +0000 UTC m=+34.264458475 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-44lhh" (UniqueName: "kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh") pod "network-check-target-8t685" (UID: "df75f0b8-d51e-4c43-a501-58dc006fef8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:17.256269 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:17.255988    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs podName:2dd23bdc-d907-4b00-95c5-24828e92da98 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:33.255979562 +0000 UTC m=+34.264475681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs") pod "network-metrics-daemon-85xds" (UID: "2dd23bdc-d907-4b00-95c5-24828e92da98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:17.538764 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:17.538668    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:17.538764 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:17.538698    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:17.538992 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:17.538670    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:17.538992 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:17.538819    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:17.538992 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:17.538894    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:17.539149 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:17.538990    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:19.539607 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:19.539575    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:19.540012 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:19.539618    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:19.540012 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:19.539680    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:19.540012 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:19.539794    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:19.540012 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:19.539834    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:19.540012 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:19.539912    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:20.720935 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.720468    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rfzkh" event={"ID":"09e1b36b-9b93-4340-a648-fafc7e0b6f90","Type":"ContainerStarted","Data":"84a00024e3d7b032027342f49966a2c0f0a2d0b075ed03a6289ad9c55f9e01a0"}
Sep 05 02:18:20.722110 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.722079    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xbqkm" event={"ID":"57d95fd2-fff8-4df2-9ae3-89d120559276","Type":"ContainerStarted","Data":"de9395ba320830f27dca18957ed64e0671cfe4f652f67e4d8bb0dd0bf17f3bde"}
Sep 05 02:18:20.723590 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.723562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-96mr6" event={"ID":"89c89ec9-f23a-4b39-83ed-d04f5c12362e","Type":"ContainerStarted","Data":"39fcdff1e2452d8c2f79ca2ec162920047e4aca87a916164969f714f98bcbcfc"}
Sep 05 02:18:20.728282 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.728261    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d" event={"ID":"d6059461-d098-4d3f-8ae4-11d5a1b09ff5","Type":"ContainerStarted","Data":"90bc3f3f7c779245b9348f43903ab211e79a0b970c79d05b8712eabbbe4417d9"}
Sep 05 02:18:20.728357 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.728337    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d" event={"ID":"d6059461-d098-4d3f-8ae4-11d5a1b09ff5","Type":"ContainerStarted","Data":"970d1d024e8af945a842be24b9111ca0e51ff24b2aea7f4136026f598343a51a"}
Sep 05 02:18:20.728417 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.728390    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d" event={"ID":"d6059461-d098-4d3f-8ae4-11d5a1b09ff5","Type":"ContainerStarted","Data":"ee4d3145e3d08e7d8abe0a1ce73c4d606dca3e194107b444cf1ddbf8bf5776ee"}
Sep 05 02:18:20.730341 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.729939    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vtds2" event={"ID":"897e2168-985a-4ac7-95eb-9eb7cf09aa12","Type":"ContainerStarted","Data":"e5dfc6378661161b790c682b89151de2360633e5dd355f526444bfee02fe078a"}
Sep 05 02:18:20.731589 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.731564    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8zqc9" event={"ID":"06c658b0-5ad2-43c5-8057-633c5525d4ab","Type":"ContainerStarted","Data":"d1fad383bc47533f022eb7fcdbaecd9704a51cec18146852502c0301ac98d005"}
Sep 05 02:18:20.733177 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.733149    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68" event={"ID":"ef7a3697-f377-4d16-afe3-c8c4108fccff","Type":"ContainerStarted","Data":"fb7e5f3c9b2d6d8362b1b8fae53b6a8cfe65cf4d3bb3620ae98bb2bbafe6cfaf"}
Sep 05 02:18:20.734914 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.734889    2555 generic.go:356] "Generic (PLEG): container finished" podID="5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae" containerID="c6f578f461872622b78fdb3eaeffc817fcd9c43cc64b4ac555a04320074dc724" exitCode=0
Sep 05 02:18:20.734988 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.734930    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb2xw" event={"ID":"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae","Type":"ContainerDied","Data":"c6f578f461872622b78fdb3eaeffc817fcd9c43cc64b4ac555a04320074dc724"}
Sep 05 02:18:20.737770 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.737708    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rfzkh" podStartSLOduration=4.004790638 podStartE2EDuration="21.73768941s" totalImagesPullingTime="17.732898772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:20.736792914 +0000 UTC m=+21.745289072" watchObservedRunningTime="2026-09-05 02:18:20.73768941 +0000 UTC m=+21.746185626"
Sep 05 02:18:20.737931 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.737886    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-239.ec2.internal" podStartSLOduration=19.737873929 podStartE2EDuration="19.737873929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:04.710021489 +0000 UTC m=+5.718517630" watchObservedRunningTime="2026-09-05 02:18:20.737873929 +0000 UTC m=+21.746370071"
Sep 05 02:18:20.771734 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.771688    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-96mr6" podStartSLOduration=11.42706109 podStartE2EDuration="21.771671738s" totalImagesPullingTime="10.344610648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:20.753513441 +0000 UTC m=+21.762009604" watchObservedRunningTime="2026-09-05 02:18:20.771671738 +0000 UTC m=+21.780167880"
Sep 05 02:18:20.771852 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.771808    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vtds2" podStartSLOduration=4.063808318 podStartE2EDuration="21.771803569s" totalImagesPullingTime="17.707995251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:20.77128833 +0000 UTC m=+21.779784469" watchObservedRunningTime="2026-09-05 02:18:20.771803569 +0000 UTC m=+21.780299708"
Sep 05 02:18:20.841786 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:20.841745    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8zqc9" podStartSLOduration=4.109038112 podStartE2EDuration="21.841729768s" totalImagesPullingTime="17.732691656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:20.813530512 +0000 UTC m=+21.822026650" watchObservedRunningTime="2026-09-05 02:18:20.841729768 +0000 UTC m=+21.850225907"
Sep 05 02:18:21.000837 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:21.000803    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96mr6_89c89ec9-f23a-4b39-83ed-d04f5c12362e/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:21.068928 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:21.068896    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96mr6_89c89ec9-f23a-4b39-83ed-d04f5c12362e/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:21.538957 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:21.538927    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:21.539180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:21.538925    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:21.539180 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:21.539039    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:21.539180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:21.538925    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:21.539180 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:21.539125    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:21.539386 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:21.539224    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:21.742113 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:21.742077    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d" event={"ID":"d6059461-d098-4d3f-8ae4-11d5a1b09ff5","Type":"ContainerStarted","Data":"483baeaefcae28b28c84f57af125382603b6ada75f46b8fabdd8487820f56e0d"}
Sep 05 02:18:21.742500 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:21.742174    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d" event={"ID":"d6059461-d098-4d3f-8ae4-11d5a1b09ff5","Type":"ContainerStarted","Data":"18c1540a7133039ed2d4073efedd3d6c4f6c961d247e4cdc586975537ef6d397"}
Sep 05 02:18:21.742500 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:21.742222    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d" event={"ID":"d6059461-d098-4d3f-8ae4-11d5a1b09ff5","Type":"ContainerStarted","Data":"8997a3180e151c26e9bd0f6f8122fb02f4fcf4711f1afc5e7f02b175360d7a39"}
Sep 05 02:18:21.864683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:21.864654    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 05 02:18:22.465443 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:22.465324    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T02:18:21.864677679Z","UUID":"487e4c43-5912-4de3-abd7-cebd6b359c5e","Handler":null,"Name":"","Endpoint":""}
Sep 05 02:18:22.467227 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:22.467201    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 05 02:18:22.467354 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:22.467235    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 05 02:18:22.747448 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:22.747367    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68" event={"ID":"ef7a3697-f377-4d16-afe3-c8c4108fccff","Type":"ContainerStarted","Data":"60ac4b3fea5ba5ff9ccdb8302d92f695ab65fa3854b55350875c80b7378b8942"}
Sep 05 02:18:22.749173 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:22.749145    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bx78n" event={"ID":"7b8c1413-db22-4560-9991-6260629f38be","Type":"ContainerStarted","Data":"47e1fe15dddc13a3abc156d24b7580823aae64b19316771b703b93a27d49116b"}
Sep 05 02:18:22.767148 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:22.767096    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xbqkm" podStartSLOduration=5.885976911 podStartE2EDuration="23.767077984s" totalImagesPullingTime="17.881101073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:20.842181821 +0000 UTC m=+21.850677960" watchObservedRunningTime="2026-09-05 02:18:22.767077984 +0000 UTC m=+23.775574126"
Sep 05 02:18:22.937622 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:22.937596    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96mr6_89c89ec9-f23a-4b39-83ed-d04f5c12362e/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:23.034524 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.034495    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96mr6_89c89ec9-f23a-4b39-83ed-d04f5c12362e/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:23.131056 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.131000    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:23.131699 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.131637    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:23.146838 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.146792    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bx78n" podStartSLOduration=6.43929685 podStartE2EDuration="24.14677999s" totalImagesPullingTime="17.70748314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:22.767233206 +0000 UTC m=+23.775729326" watchObservedRunningTime="2026-09-05 02:18:23.14677999 +0000 UTC m=+24.155276128"
Sep 05 02:18:23.538931 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.538901    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:23.539203 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.538896    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:23.539203 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.539043    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:23.539203 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:23.539033    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:23.539203 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:23.539174    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:23.539407 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:23.539273    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:23.755870 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.755836    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d" event={"ID":"d6059461-d098-4d3f-8ae4-11d5a1b09ff5","Type":"ContainerStarted","Data":"8de9033be8c91c2f7cec08b439b94051fb6659459b3dcdb06e5b8ff74f30f676"}
Sep 05 02:18:23.758240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.758207    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68" event={"ID":"ef7a3697-f377-4d16-afe3-c8c4108fccff","Type":"ContainerStarted","Data":"78dc318ec8971414690ec7046b2cca83b239d3cc1b186c72d5580321b47ef580"}
Sep 05 02:18:23.758798 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.758672    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:23.759025 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.758995    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-96mr6"
Sep 05 02:18:23.777018 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.776973    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkx68" podStartSLOduration=4.820517135 podStartE2EDuration="24.776960102s" totalImagesPullingTime="19.956442967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:23.776360739 +0000 UTC m=+24.784856897" watchObservedRunningTime="2026-09-05 02:18:23.776960102 +0000 UTC m=+24.785456240"
Sep 05 02:18:23.851830 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.851757    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96mr6_89c89ec9-f23a-4b39-83ed-d04f5c12362e/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:23.936586 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:23.936551    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96mr6_89c89ec9-f23a-4b39-83ed-d04f5c12362e/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:25.538029 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.537956    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:25.538029 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.537984    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:25.538718 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.537956    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:25.538718 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:25.538060    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:25.538718 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:25.538158    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:25.538718 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:25.538244    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:25.764292 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.764258    2555 generic.go:356] "Generic (PLEG): container finished" podID="5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae" containerID="8002df218b916d31100d5e017644f396d67771575477816560b623440b5e98fd" exitCode=0
Sep 05 02:18:25.764463 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.764342    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb2xw" event={"ID":"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae","Type":"ContainerDied","Data":"8002df218b916d31100d5e017644f396d67771575477816560b623440b5e98fd"}
Sep 05 02:18:25.768165 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.768138    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d" event={"ID":"d6059461-d098-4d3f-8ae4-11d5a1b09ff5","Type":"ContainerStarted","Data":"49ec6e88793b6032f18504ab8be9219508f2a549ada1c77998db180c38538084"}
Sep 05 02:18:25.771168 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.770506    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:25.771168 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.770627    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:25.771168 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.770694    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:25.786559 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.786520    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:25.786680 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.786630    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:18:25.817766 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:25.817681    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d" podStartSLOduration=9.093876754 podStartE2EDuration="26.817665707s" totalImagesPullingTime="17.723788953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:25.817143864 +0000 UTC m=+26.825640004" watchObservedRunningTime="2026-09-05 02:18:25.817665707 +0000 UTC m=+26.826161847"
Sep 05 02:18:27.208218 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:27.207903    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8t685"]
Sep 05 02:18:27.208632 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:27.208247    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:27.208632 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:27.208329    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:27.212046 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:27.212020    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-85xds"]
Sep 05 02:18:27.212148 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:27.212123    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:27.212218 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:27.212202    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:27.212680 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:27.212658    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8dkfv"]
Sep 05 02:18:27.212768 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:27.212757    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:27.212861 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:27.212840    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:27.774559 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:27.774516    2555 generic.go:356] "Generic (PLEG): container finished" podID="5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae" containerID="66fbd4be711fc643469cc1b391080fac755a832f6be671a4e9bd2bd79d7d6ade" exitCode=0
Sep 05 02:18:27.774723 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:27.774567    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb2xw" event={"ID":"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae","Type":"ContainerDied","Data":"66fbd4be711fc643469cc1b391080fac755a832f6be671a4e9bd2bd79d7d6ade"}
Sep 05 02:18:28.538392 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:28.538354    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:28.538810 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:28.538361    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:28.538810 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:28.538498    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:28.538810 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:28.538615    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:29.538873 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:29.538844    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:29.539329 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:29.538957    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:29.782184 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:29.781958    2555 generic.go:356] "Generic (PLEG): container finished" podID="5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae" containerID="9c8ec8ee4e5ab887c4d3b8c8073ecc9b540cae1d9c050bec67534e09cb607e24" exitCode=0
Sep 05 02:18:29.782184 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:29.782043    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb2xw" event={"ID":"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae","Type":"ContainerDied","Data":"9c8ec8ee4e5ab887c4d3b8c8073ecc9b540cae1d9c050bec67534e09cb607e24"}
Sep 05 02:18:30.538638 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:30.538605    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:30.538803 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:30.538605    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:30.538803 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:30.538726    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:30.538803 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:30.538774    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:31.538200 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:31.538131    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:31.538431 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:31.538270    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85xds" podUID="2dd23bdc-d907-4b00-95c5-24828e92da98"
Sep 05 02:18:32.538093 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:32.538063    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:32.538524 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:32.538073    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:32.538524 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:32.538241    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dkfv" podUID="bd509ebe-1474-4b42-88ae-96220ceecee8"
Sep 05 02:18:32.538524 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:32.538338    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8t685" podUID="df75f0b8-d51e-4c43-a501-58dc006fef8e"
Sep 05 02:18:33.254259 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.254183    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-239.ec2.internal" event="NodeReady"
Sep 05 02:18:33.254429 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.254346    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 05 02:18:33.268030 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.268007    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lhh\" (UniqueName: \"kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh\") pod \"network-check-target-8t685\" (UID: \"df75f0b8-d51e-4c43-a501-58dc006fef8e\") " pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:33.268141 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.268041    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:33.268141 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.268064    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:33.268237 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:33.268162    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:33.268237 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:33.268177    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:33.268237 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:33.268186    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:33.268237 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:33.268196    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:33.268237 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:33.268209    2555 projected.go:196] Error preparing data for projected volume kube-api-access-44lhh for pod openshift-network-diagnostics/network-check-target-8t685: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:33.268237 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:33.268221    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs podName:2dd23bdc-d907-4b00-95c5-24828e92da98 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:05.268203195 +0000 UTC m=+66.276699325 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs") pod "network-metrics-daemon-85xds" (UID: "2dd23bdc-d907-4b00-95c5-24828e92da98") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:33.268477 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:33.268241    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret podName:bd509ebe-1474-4b42-88ae-96220ceecee8 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:05.268227025 +0000 UTC m=+66.276723145 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret") pod "global-pull-secret-syncer-8dkfv" (UID: "bd509ebe-1474-4b42-88ae-96220ceecee8") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:33.268477 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:33.268256    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh podName:df75f0b8-d51e-4c43-a501-58dc006fef8e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:05.268249925 +0000 UTC m=+66.276746045 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-44lhh" (UniqueName: "kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh") pod "network-check-target-8t685" (UID: "df75f0b8-d51e-4c43-a501-58dc006fef8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:33.316930 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.316902    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9l9d9"]
Sep 05 02:18:33.322001 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.321980    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rsph8"]
Sep 05 02:18:33.322133 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.322115    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.324731 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.324698    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 05 02:18:33.324838 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.324722    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2swhn\""
Sep 05 02:18:33.324838 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.324822    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 05 02:18:33.325170 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.325152    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rsph8"
Sep 05 02:18:33.328029 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.328011    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 05 02:18:33.328393 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.328280    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 05 02:18:33.328393 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.328335    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 05 02:18:33.328576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.328561    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dnzd9\""
Sep 05 02:18:33.330556 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.330504    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rsph8"]
Sep 05 02:18:33.333187 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.333164    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9l9d9"]
Sep 05 02:18:33.470006 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.469968    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.470203 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.470017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj48w\" (UniqueName: \"kubernetes.io/projected/4f3f8d4a-ee71-418d-958e-3a92b0d250be-kube-api-access-cj48w\") pod \"ingress-canary-rsph8\" (UID: \"4f3f8d4a-ee71-418d-958e-3a92b0d250be\") " pod="openshift-ingress-canary/ingress-canary-rsph8"
Sep 05 02:18:33.470203 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.470093    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ptt7\" (UniqueName: \"kubernetes.io/projected/f0769ef0-90d9-43ee-ade4-5aae6141a935-kube-api-access-4ptt7\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.470323 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.470228    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0769ef0-90d9-43ee-ade4-5aae6141a935-config-volume\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.470323 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.470260    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0769ef0-90d9-43ee-ade4-5aae6141a935-tmp-dir\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.470323 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.470294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f3f8d4a-ee71-418d-958e-3a92b0d250be-cert\") pod \"ingress-canary-rsph8\" (UID: \"4f3f8d4a-ee71-418d-958e-3a92b0d250be\") " pod="openshift-ingress-canary/ingress-canary-rsph8"
Sep 05 02:18:33.541907 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.541835    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:18:33.544619 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.544592    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:33.544745 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.544628    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ptsx2\""
Sep 05 02:18:33.570758 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.570723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0769ef0-90d9-43ee-ade4-5aae6141a935-config-volume\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.570758 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.570762    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0769ef0-90d9-43ee-ade4-5aae6141a935-tmp-dir\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.570990 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.570781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f3f8d4a-ee71-418d-958e-3a92b0d250be-cert\") pod \"ingress-canary-rsph8\" (UID: \"4f3f8d4a-ee71-418d-958e-3a92b0d250be\") " pod="openshift-ingress-canary/ingress-canary-rsph8"
Sep 05 02:18:33.570990 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.570817    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.570990 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:33.570908    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:33.570990 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.570926    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj48w\" (UniqueName: \"kubernetes.io/projected/4f3f8d4a-ee71-418d-958e-3a92b0d250be-kube-api-access-cj48w\") pod \"ingress-canary-rsph8\" (UID: \"4f3f8d4a-ee71-418d-958e-3a92b0d250be\") " pod="openshift-ingress-canary/ingress-canary-rsph8"
Sep 05 02:18:33.570990 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:33.570960    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls podName:f0769ef0-90d9-43ee-ade4-5aae6141a935 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:34.070944311 +0000 UTC m=+35.079440427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls") pod "dns-default-9l9d9" (UID: "f0769ef0-90d9-43ee-ade4-5aae6141a935") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:33.571216 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.570996    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ptt7\" (UniqueName: \"kubernetes.io/projected/f0769ef0-90d9-43ee-ade4-5aae6141a935-kube-api-access-4ptt7\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.571216 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.571146    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0769ef0-90d9-43ee-ade4-5aae6141a935-tmp-dir\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.571420 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.571393    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0769ef0-90d9-43ee-ade4-5aae6141a935-config-volume\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.575348 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.575330    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f3f8d4a-ee71-418d-958e-3a92b0d250be-cert\") pod \"ingress-canary-rsph8\" (UID: \"4f3f8d4a-ee71-418d-958e-3a92b0d250be\") " pod="openshift-ingress-canary/ingress-canary-rsph8"
Sep 05 02:18:33.583304 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.583279    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj48w\" (UniqueName: \"kubernetes.io/projected/4f3f8d4a-ee71-418d-958e-3a92b0d250be-kube-api-access-cj48w\") pod \"ingress-canary-rsph8\" (UID: \"4f3f8d4a-ee71-418d-958e-3a92b0d250be\") " pod="openshift-ingress-canary/ingress-canary-rsph8"
Sep 05 02:18:33.584524 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.584502    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ptt7\" (UniqueName: \"kubernetes.io/projected/f0769ef0-90d9-43ee-ade4-5aae6141a935-kube-api-access-4ptt7\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:33.641678 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.641651    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rsph8"
Sep 05 02:18:33.942858 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:33.942693    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rsph8"]
Sep 05 02:18:33.945788 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:18:33.945757    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3f8d4a_ee71_418d_958e_3a92b0d250be.slice/crio-bccb61022d01d18ffeb94809ab70e168777e408832976da46d774bc2292b763d WatchSource:0}: Error finding container bccb61022d01d18ffeb94809ab70e168777e408832976da46d774bc2292b763d: Status 404 returned error can't find the container with id bccb61022d01d18ffeb94809ab70e168777e408832976da46d774bc2292b763d
Sep 05 02:18:34.074566 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:34.074465    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:34.074699 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:34.074621    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:34.074699 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:34.074688    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls podName:f0769ef0-90d9-43ee-ade4-5aae6141a935 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:35.074668817 +0000 UTC m=+36.083164950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls") pod "dns-default-9l9d9" (UID: "f0769ef0-90d9-43ee-ade4-5aae6141a935") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:34.538562 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:34.538437    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:18:34.538562 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:34.538437    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:18:34.541501 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:34.541467    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:34.541501 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:34.541485    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:34.541501 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:34.541491    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:34.541736 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:34.541563    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fnrtr\""
Sep 05 02:18:34.797898 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:34.797813    2555 generic.go:356] "Generic (PLEG): container finished" podID="5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae" containerID="5b7186a44ed4f25d9a1f4b995f541fdcfa7356020a04576f475bb5676d08cae7" exitCode=0
Sep 05 02:18:34.798529 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:34.797892    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb2xw" event={"ID":"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae","Type":"ContainerDied","Data":"5b7186a44ed4f25d9a1f4b995f541fdcfa7356020a04576f475bb5676d08cae7"}
Sep 05 02:18:34.799629 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:34.799603    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rsph8" event={"ID":"4f3f8d4a-ee71-418d-958e-3a92b0d250be","Type":"ContainerStarted","Data":"bccb61022d01d18ffeb94809ab70e168777e408832976da46d774bc2292b763d"}
Sep 05 02:18:35.081960 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:35.081880    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:35.082128 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:35.082022    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:35.082128 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:35.082096    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls podName:f0769ef0-90d9-43ee-ade4-5aae6141a935 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:37.082076384 +0000 UTC m=+38.090572517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls") pod "dns-default-9l9d9" (UID: "f0769ef0-90d9-43ee-ade4-5aae6141a935") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:35.805573 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:35.805518    2555 generic.go:356] "Generic (PLEG): container finished" podID="5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae" containerID="8cffcda89c62678dbea529c4b31fadf4e419ae41d26b5bfc0b00d640ab65bf31" exitCode=0
Sep 05 02:18:35.805998 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:35.805570    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb2xw" event={"ID":"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae","Type":"ContainerDied","Data":"8cffcda89c62678dbea529c4b31fadf4e419ae41d26b5bfc0b00d640ab65bf31"}
Sep 05 02:18:36.811451 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:36.811408    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb2xw" event={"ID":"5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae","Type":"ContainerStarted","Data":"1d606a8230dbef7529e7a861b1f0f86c3a559c2b5086204fb8781f8bd825175d"}
Sep 05 02:18:36.812920 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:36.812897    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rsph8" event={"ID":"4f3f8d4a-ee71-418d-958e-3a92b0d250be","Type":"ContainerStarted","Data":"f621e0312f40701a662efbe81ab96c785e3740c4de6d3da32a3f694ea2868370"}
Sep 05 02:18:36.835217 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:36.835169    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qb2xw" podStartSLOduration=9.767415418 podStartE2EDuration="37.835154868s" totalImagesPullingTime="28.06773945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:36.834133939 +0000 UTC m=+37.842630078" watchObservedRunningTime="2026-09-05 02:18:36.835154868 +0000 UTC m=+37.843651008"
Sep 05 02:18:36.849932 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:36.849893    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rsph8" podStartSLOduration=1.33293521 podStartE2EDuration="3.84987743s" totalImagesPullingTime="2.51694222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:36.849358456 +0000 UTC m=+37.857854596" watchObservedRunningTime="2026-09-05 02:18:36.84987743 +0000 UTC m=+37.858373568"
Sep 05 02:18:37.097620 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:37.097510    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:37.097765 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:37.097639    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:37.097765 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:37.097709    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls podName:f0769ef0-90d9-43ee-ade4-5aae6141a935 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:41.097690604 +0000 UTC m=+42.106186726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls") pod "dns-default-9l9d9" (UID: "f0769ef0-90d9-43ee-ade4-5aae6141a935") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:41.124141 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:41.123948    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:41.124141 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:41.124113    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:41.124585 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:41.124210    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls podName:f0769ef0-90d9-43ee-ade4-5aae6141a935 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:49.12419481 +0000 UTC m=+50.132690927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls") pod "dns-default-9l9d9" (UID: "f0769ef0-90d9-43ee-ade4-5aae6141a935") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:49.184732 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:49.184690    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:18:49.185182 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:49.184844    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:49.185182 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:18:49.184911    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls podName:f0769ef0-90d9-43ee-ade4-5aae6141a935 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:05.184894257 +0000 UTC m=+66.193390394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls") pod "dns-default-9l9d9" (UID: "f0769ef0-90d9-43ee-ade4-5aae6141a935") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:57.794524 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:18:57.794488    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2sg7d"
Sep 05 02:19:03.435524 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.435482    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l9d7p"]
Sep 05 02:19:03.438594 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.438576    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l9d7p"
Sep 05 02:19:03.442978 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.442959    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:19:03.443200 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.443180    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ztrv5\""
Sep 05 02:19:03.443283 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.443183    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 05 02:19:03.448771 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.448752    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l9d7p"]
Sep 05 02:19:03.482506 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.482471    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48m9k\" (UniqueName: \"kubernetes.io/projected/ba09a841-916f-435d-8f70-221248e93a65-kube-api-access-48m9k\") pod \"volume-data-source-validator-695c895bd6-l9d7p\" (UID: \"ba09a841-916f-435d-8f70-221248e93a65\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l9d7p"
Sep 05 02:19:03.526227 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.526194    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vwlqs"]
Sep 05 02:19:03.528014 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.527998    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.533731 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.533710    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 05 02:19:03.533731 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.533726    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 05 02:19:03.533731 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.533750    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 05 02:19:03.534133 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.534028    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:19:03.534336 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.534320    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-glv8t\""
Sep 05 02:19:03.559484 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.559454    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vwlqs"]
Sep 05 02:19:03.561815 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.561795    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 05 02:19:03.583236 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.583205    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-serving-cert\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.583362 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.583295    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48m9k\" (UniqueName: \"kubernetes.io/projected/ba09a841-916f-435d-8f70-221248e93a65-kube-api-access-48m9k\") pod \"volume-data-source-validator-695c895bd6-l9d7p\" (UID: \"ba09a841-916f-435d-8f70-221248e93a65\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l9d7p"
Sep 05 02:19:03.583362 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.583322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5wg8\" (UniqueName: \"kubernetes.io/projected/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-kube-api-access-f5wg8\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.583362 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.583347    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-trusted-ca\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.583502 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.583372    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-config\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.601863 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.601839    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48m9k\" (UniqueName: \"kubernetes.io/projected/ba09a841-916f-435d-8f70-221248e93a65-kube-api-access-48m9k\") pod \"volume-data-source-validator-695c895bd6-l9d7p\" (UID: \"ba09a841-916f-435d-8f70-221248e93a65\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l9d7p"
Sep 05 02:19:03.684148 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.684114    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5wg8\" (UniqueName: \"kubernetes.io/projected/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-kube-api-access-f5wg8\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.684307 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.684153    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-trusted-ca\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.684307 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.684177    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-config\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.684307 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.684231    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-serving-cert\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.684886 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.684863    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-config\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.684985 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.684968    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-trusted-ca\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.686606 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.686558    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-serving-cert\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.688967 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.688945    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-cbsx2"]
Sep 05 02:19:03.690997 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.690980    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-cbsx2"
Sep 05 02:19:03.693840 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.693821    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-29wlj\""
Sep 05 02:19:03.736643 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.736616    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-697d84c94b-6j7tj"]
Sep 05 02:19:03.738568 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.738505    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5wg8\" (UniqueName: \"kubernetes.io/projected/30f1f819-6e52-4eb7-b546-47f22a4d5f8b-kube-api-access-f5wg8\") pod \"console-operator-85b454874b-vwlqs\" (UID: \"30f1f819-6e52-4eb7-b546-47f22a4d5f8b\") " pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.738723 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.738707    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-cbsx2"]
Sep 05 02:19:03.738809 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.738798    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.741722 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.741701    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 05 02:19:03.747151 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.747134    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l9d7p"
Sep 05 02:19:03.749769 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.749750    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 05 02:19:03.749896 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.749783    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 05 02:19:03.750848 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.750810    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7th2q\""
Sep 05 02:19:03.761113 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.761088    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 05 02:19:03.784742 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.784709    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-trusted-ca\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.784916 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.784754    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-image-registry-private-configuration\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.784916 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.784782    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-installation-pull-secrets\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.784916 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.784851    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1edddb2-de81-436c-bd1b-720e994f6e70-ca-trust-extracted\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.785113 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.784931    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-certificates\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.785113 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.784956    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8nct\" (UniqueName: \"kubernetes.io/projected/0664d7c3-d7ef-41b5-97f5-8e9fd173da99-kube-api-access-r8nct\") pod \"network-check-source-685456655f-cbsx2\" (UID: \"0664d7c3-d7ef-41b5-97f5-8e9fd173da99\") " pod="openshift-network-diagnostics/network-check-source-685456655f-cbsx2"
Sep 05 02:19:03.785113 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.784985    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.785113 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.785015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6mnp\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-kube-api-access-n6mnp\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.785402 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.785233    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-bound-sa-token\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.793123 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.793063    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-697d84c94b-6j7tj"]
Sep 05 02:19:03.836778 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.836752    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:03.885983 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.885958    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-bound-sa-token\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.886071 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.885993    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-trusted-ca\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.886071 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.886014    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-image-registry-private-configuration\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.886071 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.886032    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-installation-pull-secrets\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.886071 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.886047    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1edddb2-de81-436c-bd1b-720e994f6e70-ca-trust-extracted\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.886200 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.886084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-certificates\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.886200 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.886098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8nct\" (UniqueName: \"kubernetes.io/projected/0664d7c3-d7ef-41b5-97f5-8e9fd173da99-kube-api-access-r8nct\") pod \"network-check-source-685456655f-cbsx2\" (UID: \"0664d7c3-d7ef-41b5-97f5-8e9fd173da99\") " pod="openshift-network-diagnostics/network-check-source-685456655f-cbsx2"
Sep 05 02:19:03.886200 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.886117    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.886200 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.886133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6mnp\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-kube-api-access-n6mnp\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.886805 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.886709    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1edddb2-de81-436c-bd1b-720e994f6e70-ca-trust-extracted\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.887285 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:03.887003    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:03.887285 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:03.887025    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-697d84c94b-6j7tj: secret "image-registry-tls" not found
Sep 05 02:19:03.887285 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:03.887079    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls podName:b1edddb2-de81-436c-bd1b-720e994f6e70 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:04.387059882 +0000 UTC m=+65.395556002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls") pod "image-registry-697d84c94b-6j7tj" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70") : secret "image-registry-tls" not found
Sep 05 02:19:03.887483 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.887367    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-certificates\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.887604 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.887499    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-trusted-ca\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.889193 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.889147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-image-registry-private-configuration\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.890388 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.890268    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-installation-pull-secrets\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.899506 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.899481    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-bound-sa-token\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.899884 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.899861    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6mnp\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-kube-api-access-n6mnp\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:03.900348 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.900331    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:19:03.910442 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.910418    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:19:03.920618 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.920599    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8nct\" (UniqueName: \"kubernetes.io/projected/0664d7c3-d7ef-41b5-97f5-8e9fd173da99-kube-api-access-r8nct\") pod \"network-check-source-685456655f-cbsx2\" (UID: \"0664d7c3-d7ef-41b5-97f5-8e9fd173da99\") " pod="openshift-network-diagnostics/network-check-source-685456655f-cbsx2"
Sep 05 02:19:03.999246 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:03.999182    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-cbsx2"
Sep 05 02:19:04.015721 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:04.015695    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l9d7p"]
Sep 05 02:19:04.018463 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:04.018439    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba09a841_916f_435d_8f70_221248e93a65.slice/crio-b13db6d796125bbf939562d7730d9af7944b848dd3225e0e9c53e058c48124e1 WatchSource:0}: Error finding container b13db6d796125bbf939562d7730d9af7944b848dd3225e0e9c53e058c48124e1: Status 404 returned error can't find the container with id b13db6d796125bbf939562d7730d9af7944b848dd3225e0e9c53e058c48124e1
Sep 05 02:19:04.108245 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:04.108214    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vwlqs"]
Sep 05 02:19:04.110934 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:04.110908    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f1f819_6e52_4eb7_b546_47f22a4d5f8b.slice/crio-b587aaa07c54b196fa6f11119ca81ff42b7cd9fe7b09fc7991fe415fcd007cd6 WatchSource:0}: Error finding container b587aaa07c54b196fa6f11119ca81ff42b7cd9fe7b09fc7991fe415fcd007cd6: Status 404 returned error can't find the container with id b587aaa07c54b196fa6f11119ca81ff42b7cd9fe7b09fc7991fe415fcd007cd6
Sep 05 02:19:04.265951 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:04.265878    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-cbsx2"]
Sep 05 02:19:04.268690 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:04.268664    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0664d7c3_d7ef_41b5_97f5_8e9fd173da99.slice/crio-6f4c2e9f0d11e580abb3b66d702d674d62795e77798ac36f4f1510e86f3f3fbb WatchSource:0}: Error finding container 6f4c2e9f0d11e580abb3b66d702d674d62795e77798ac36f4f1510e86f3f3fbb: Status 404 returned error can't find the container with id 6f4c2e9f0d11e580abb3b66d702d674d62795e77798ac36f4f1510e86f3f3fbb
Sep 05 02:19:04.391016 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:04.390982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:04.391162 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:04.391150    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:04.391223 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:04.391167    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-697d84c94b-6j7tj: secret "image-registry-tls" not found
Sep 05 02:19:04.391262 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:04.391224    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls podName:b1edddb2-de81-436c-bd1b-720e994f6e70 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:05.39120819 +0000 UTC m=+66.399704307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls") pod "image-registry-697d84c94b-6j7tj" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70") : secret "image-registry-tls" not found
Sep 05 02:19:04.878471 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:04.878396    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-cbsx2" event={"ID":"0664d7c3-d7ef-41b5-97f5-8e9fd173da99","Type":"ContainerStarted","Data":"6f4c2e9f0d11e580abb3b66d702d674d62795e77798ac36f4f1510e86f3f3fbb"}
Sep 05 02:19:04.879844 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:04.879802    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vwlqs" event={"ID":"30f1f819-6e52-4eb7-b546-47f22a4d5f8b","Type":"ContainerStarted","Data":"b587aaa07c54b196fa6f11119ca81ff42b7cd9fe7b09fc7991fe415fcd007cd6"}
Sep 05 02:19:04.881118 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:04.881089    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l9d7p" event={"ID":"ba09a841-916f-435d-8f70-221248e93a65","Type":"ContainerStarted","Data":"b13db6d796125bbf939562d7730d9af7944b848dd3225e0e9c53e058c48124e1"}
Sep 05 02:19:05.198408 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.198311    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:19:05.198597 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:05.198487    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:05.198597 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:05.198580    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls podName:f0769ef0-90d9-43ee-ade4-5aae6141a935 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:37.198560326 +0000 UTC m=+98.207056443 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls") pod "dns-default-9l9d9" (UID: "f0769ef0-90d9-43ee-ade4-5aae6141a935") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:05.299354 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.299313    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lhh\" (UniqueName: \"kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh\") pod \"network-check-target-8t685\" (UID: \"df75f0b8-d51e-4c43-a501-58dc006fef8e\") " pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:19:05.299549 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.299398    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:19:05.299549 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.299429    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:19:05.302158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.302048    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:19:05.302577 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.302426    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44lhh\" (UniqueName: \"kubernetes.io/projected/df75f0b8-d51e-4c43-a501-58dc006fef8e-kube-api-access-44lhh\") pod \"network-check-target-8t685\" (UID: \"df75f0b8-d51e-4c43-a501-58dc006fef8e\") " pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:19:05.302577 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.302456    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:19:05.309776 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:05.309747    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 02:19:05.309869 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:05.309824    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs podName:2dd23bdc-d907-4b00-95c5-24828e92da98 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:09.309801335 +0000 UTC m=+130.318297464 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs") pod "network-metrics-daemon-85xds" (UID: "2dd23bdc-d907-4b00-95c5-24828e92da98") : secret "metrics-daemon-secret" not found
Sep 05 02:19:05.312022 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.312000    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd509ebe-1474-4b42-88ae-96220ceecee8-original-pull-secret\") pod \"global-pull-secret-syncer-8dkfv\" (UID: \"bd509ebe-1474-4b42-88ae-96220ceecee8\") " pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:19:05.400128 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.400080    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:05.400285 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:05.400229    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:05.400285 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:05.400251    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-697d84c94b-6j7tj: secret "image-registry-tls" not found
Sep 05 02:19:05.400404 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:05.400318    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls podName:b1edddb2-de81-436c-bd1b-720e994f6e70 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:07.40029852 +0000 UTC m=+68.408794657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls") pod "image-registry-697d84c94b-6j7tj" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70") : secret "image-registry-tls" not found
Sep 05 02:19:05.450164 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.450106    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fnrtr\""
Sep 05 02:19:05.452125 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.452100    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dkfv"
Sep 05 02:19:05.457795 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.457774    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:19:05.758297 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.758173    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8dkfv"]
Sep 05 02:19:05.775567 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.775483    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8t685"]
Sep 05 02:19:05.886924 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.886891    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l9d7p" event={"ID":"ba09a841-916f-435d-8f70-221248e93a65","Type":"ContainerStarted","Data":"fcc12caff940a2845e81f7a28916a98fea78f83561603ec9bc35fb9357219aba"}
Sep 05 02:19:05.903692 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:05.903648    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l9d7p" podStartSLOduration=1.576026508 podStartE2EDuration="2.903627728s" totalImagesPullingTime="1.32760122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:05.902459246 +0000 UTC m=+66.910955409" watchObservedRunningTime="2026-09-05 02:19:05.903627728 +0000 UTC m=+66.912123867"
Sep 05 02:19:06.291731 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:06.291689    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd509ebe_1474_4b42_88ae_96220ceecee8.slice/crio-4ac6196baf3b995ebdaab741c08f2dcff2710c3b1c64c0a6c0c90dfedb1b20c1 WatchSource:0}: Error finding container 4ac6196baf3b995ebdaab741c08f2dcff2710c3b1c64c0a6c0c90dfedb1b20c1: Status 404 returned error can't find the container with id 4ac6196baf3b995ebdaab741c08f2dcff2710c3b1c64c0a6c0c90dfedb1b20c1
Sep 05 02:19:06.292319 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:06.292255    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf75f0b8_d51e_4c43_a501_58dc006fef8e.slice/crio-ad6cf240eaa5e82c324568f7cdc63014d48fba41c398ead9f641247dc9b5ea00 WatchSource:0}: Error finding container ad6cf240eaa5e82c324568f7cdc63014d48fba41c398ead9f641247dc9b5ea00: Status 404 returned error can't find the container with id ad6cf240eaa5e82c324568f7cdc63014d48fba41c398ead9f641247dc9b5ea00
Sep 05 02:19:06.892910 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:06.892841    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8dkfv" event={"ID":"bd509ebe-1474-4b42-88ae-96220ceecee8","Type":"ContainerStarted","Data":"4ac6196baf3b995ebdaab741c08f2dcff2710c3b1c64c0a6c0c90dfedb1b20c1"}
Sep 05 02:19:06.895339 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:06.895312    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/0.log"
Sep 05 02:19:06.895468 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:06.895369    2555 generic.go:356] "Generic (PLEG): container finished" podID="30f1f819-6e52-4eb7-b546-47f22a4d5f8b" containerID="c6bf2bb44f85ba2bcb83e2d1272699894c6fb27c68472ba6339129b18171913f" exitCode=255
Sep 05 02:19:06.895468 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:06.895402    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vwlqs" event={"ID":"30f1f819-6e52-4eb7-b546-47f22a4d5f8b","Type":"ContainerDied","Data":"c6bf2bb44f85ba2bcb83e2d1272699894c6fb27c68472ba6339129b18171913f"}
Sep 05 02:19:06.895731 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:06.895706    2555 scope.go:117] "RemoveContainer" containerID="c6bf2bb44f85ba2bcb83e2d1272699894c6fb27c68472ba6339129b18171913f"
Sep 05 02:19:06.897048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:06.897018    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8t685" event={"ID":"df75f0b8-d51e-4c43-a501-58dc006fef8e","Type":"ContainerStarted","Data":"ad6cf240eaa5e82c324568f7cdc63014d48fba41c398ead9f641247dc9b5ea00"}
Sep 05 02:19:07.415987 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:07.415317    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:07.415987 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:07.415547    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:07.415987 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:07.415566    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-697d84c94b-6j7tj: secret "image-registry-tls" not found
Sep 05 02:19:07.415987 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:07.415628    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls podName:b1edddb2-de81-436c-bd1b-720e994f6e70 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:11.415608859 +0000 UTC m=+72.424104980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls") pod "image-registry-697d84c94b-6j7tj" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70") : secret "image-registry-tls" not found
Sep 05 02:19:08.078469 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.078430    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-x6s4p"]
Sep 05 02:19:08.081688 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.081665    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:08.084311 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.084285    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 05 02:19:08.084866 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.084623    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pvmlt\""
Sep 05 02:19:08.085369 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.085153    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 05 02:19:08.095004 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.094979    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-x6s4p"]
Sep 05 02:19:08.121821 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.121792    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6s4p\" (UID: \"b0178d25-ade8-4d57-bacc-137fc466bbce\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:08.121956 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.121849    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0178d25-ade8-4d57-bacc-137fc466bbce-nginx-conf\") pod \"networking-console-plugin-79977f9844-x6s4p\" (UID: \"b0178d25-ade8-4d57-bacc-137fc466bbce\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:08.223055 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.222592    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6s4p\" (UID: \"b0178d25-ade8-4d57-bacc-137fc466bbce\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:08.223055 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.222678    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0178d25-ade8-4d57-bacc-137fc466bbce-nginx-conf\") pod \"networking-console-plugin-79977f9844-x6s4p\" (UID: \"b0178d25-ade8-4d57-bacc-137fc466bbce\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:08.223278 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:08.223154    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 05 02:19:08.223278 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:08.223232    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert podName:b0178d25-ade8-4d57-bacc-137fc466bbce nodeName:}" failed. No retries permitted until 2026-09-05 02:19:08.723211431 +0000 UTC m=+69.731707570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6s4p" (UID: "b0178d25-ade8-4d57-bacc-137fc466bbce") : secret "networking-console-plugin-cert" not found
Sep 05 02:19:08.223492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.223461    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0178d25-ade8-4d57-bacc-137fc466bbce-nginx-conf\") pod \"networking-console-plugin-79977f9844-x6s4p\" (UID: \"b0178d25-ade8-4d57-bacc-137fc466bbce\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:08.726927 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.726884    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6s4p\" (UID: \"b0178d25-ade8-4d57-bacc-137fc466bbce\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:08.727148 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:08.727034    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 05 02:19:08.727148 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:08.727118    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert podName:b0178d25-ade8-4d57-bacc-137fc466bbce nodeName:}" failed. No retries permitted until 2026-09-05 02:19:09.727096631 +0000 UTC m=+70.735592763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6s4p" (UID: "b0178d25-ade8-4d57-bacc-137fc466bbce") : secret "networking-console-plugin-cert" not found
Sep 05 02:19:08.909090 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.909054    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-cbsx2" event={"ID":"0664d7c3-d7ef-41b5-97f5-8e9fd173da99","Type":"ContainerStarted","Data":"11e0666cb504477ba3ebad0b2b45d641face14683abbda9f4641391107d66c2d"}
Sep 05 02:19:08.910648 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.910626    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:19:08.911035 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.911020    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/0.log"
Sep 05 02:19:08.911084 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.911051    2555 generic.go:356] "Generic (PLEG): container finished" podID="30f1f819-6e52-4eb7-b546-47f22a4d5f8b" containerID="922ec9d3f2feb90421b857683188a0b321af3a3a26fd6d67ecc1d00ab9dde578" exitCode=255
Sep 05 02:19:08.911132 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.911115    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vwlqs" event={"ID":"30f1f819-6e52-4eb7-b546-47f22a4d5f8b","Type":"ContainerDied","Data":"922ec9d3f2feb90421b857683188a0b321af3a3a26fd6d67ecc1d00ab9dde578"}
Sep 05 02:19:08.911183 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.911165    2555 scope.go:117] "RemoveContainer" containerID="c6bf2bb44f85ba2bcb83e2d1272699894c6fb27c68472ba6339129b18171913f"
Sep 05 02:19:08.911336 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.911317    2555 scope.go:117] "RemoveContainer" containerID="922ec9d3f2feb90421b857683188a0b321af3a3a26fd6d67ecc1d00ab9dde578"
Sep 05 02:19:08.911510 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:08.911490    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vwlqs_openshift-console-operator(30f1f819-6e52-4eb7-b546-47f22a4d5f8b)\"" pod="openshift-console-operator/console-operator-85b454874b-vwlqs" podUID="30f1f819-6e52-4eb7-b546-47f22a4d5f8b"
Sep 05 02:19:08.912642 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.912618    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8t685" event={"ID":"df75f0b8-d51e-4c43-a501-58dc006fef8e","Type":"ContainerStarted","Data":"0eb8d7655e2ec4180f6698c9323f29e96ca408d918b6e8ff2209edb766140935"}
Sep 05 02:19:08.912762 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.912748    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:19:08.940408 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.940361    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-cbsx2" podStartSLOduration=1.914152377 podStartE2EDuration="5.940344563s" totalImagesPullingTime="4.026192186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:08.939360577 +0000 UTC m=+69.947856715" watchObservedRunningTime="2026-09-05 02:19:08.940344563 +0000 UTC m=+69.948840694"
Sep 05 02:19:08.964404 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.964347    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8t685" podStartSLOduration=67.722651664 podStartE2EDuration="1m9.96432865s" totalImagesPullingTime="2.241676986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:08.962432632 +0000 UTC m=+69.970928797" watchObservedRunningTime="2026-09-05 02:19:08.96432865 +0000 UTC m=+69.972824791"
Sep 05 02:19:08.993485 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.993451    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4cllv"]
Sep 05 02:19:08.998072 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:08.998050    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.000973 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.000951    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 05 02:19:09.001195 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.001177    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 05 02:19:09.001492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.001318    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-twl8z\""
Sep 05 02:19:09.001492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.001445    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 05 02:19:09.003135 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.003118    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 05 02:19:09.013494 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.013475    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4cllv"]
Sep 05 02:19:09.029641 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.029618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.029752 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.029736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-data-volume\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.029812 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.029780    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.029812 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.029798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nl4v\" (UniqueName: \"kubernetes.io/projected/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-api-access-9nl4v\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.029902 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.029817    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-crio-socket\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.130254 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.130220    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.130696 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.130340    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-data-volume\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.130696 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:09.130367    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:09.130696 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.130394    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.130696 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.130421    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nl4v\" (UniqueName: \"kubernetes.io/projected/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-api-access-9nl4v\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.130696 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:09.130451    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm podName:6b2c3ca8-5f11-4fc2-a88c-70e8a498d693 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:09.630429685 +0000 UTC m=+70.638925809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4cllv" (UID: "6b2c3ca8-5f11-4fc2-a88c-70e8a498d693") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:09.130696 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.130488    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-crio-socket\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.130696 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:09.130509    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:09.130696 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:09.130588    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls podName:6b2c3ca8-5f11-4fc2-a88c-70e8a498d693 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:09.630571014 +0000 UTC m=+70.639067140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4cllv" (UID: "6b2c3ca8-5f11-4fc2-a88c-70e8a498d693") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:09.130696 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.130593    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-crio-socket\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.131066 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.130734    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-data-volume\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.145488 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.145450    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nl4v\" (UniqueName: \"kubernetes.io/projected/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-api-access-9nl4v\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.635987 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.635950    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.636175 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:09.636093    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:09.636239 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:09.636183    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm podName:6b2c3ca8-5f11-4fc2-a88c-70e8a498d693 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:10.636160157 +0000 UTC m=+71.644656276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4cllv" (UID: "6b2c3ca8-5f11-4fc2-a88c-70e8a498d693") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:09.636239 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.636212    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:09.636348 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:09.636334    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:09.636386 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:09.636378    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls podName:6b2c3ca8-5f11-4fc2-a88c-70e8a498d693 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:10.636367986 +0000 UTC m=+71.644864103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4cllv" (UID: "6b2c3ca8-5f11-4fc2-a88c-70e8a498d693") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:09.737516 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.737470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6s4p\" (UID: \"b0178d25-ade8-4d57-bacc-137fc466bbce\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:09.737708 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:09.737659    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 05 02:19:09.737768 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:09.737745    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert podName:b0178d25-ade8-4d57-bacc-137fc466bbce nodeName:}" failed. No retries permitted until 2026-09-05 02:19:11.737722308 +0000 UTC m=+72.746218429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6s4p" (UID: "b0178d25-ade8-4d57-bacc-137fc466bbce") : secret "networking-console-plugin-cert" not found
Sep 05 02:19:09.917196 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:09.917110    2555 scope.go:117] "RemoveContainer" containerID="922ec9d3f2feb90421b857683188a0b321af3a3a26fd6d67ecc1d00ab9dde578"
Sep 05 02:19:09.917355 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:09.917272    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vwlqs_openshift-console-operator(30f1f819-6e52-4eb7-b546-47f22a4d5f8b)\"" pod="openshift-console-operator/console-operator-85b454874b-vwlqs" podUID="30f1f819-6e52-4eb7-b546-47f22a4d5f8b"
Sep 05 02:19:10.644961 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:10.644931    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:10.645332 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:10.645048    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:10.645332 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:10.645058    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:10.645332 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:10.645125    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm podName:6b2c3ca8-5f11-4fc2-a88c-70e8a498d693 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:12.645107725 +0000 UTC m=+73.653603844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4cllv" (UID: "6b2c3ca8-5f11-4fc2-a88c-70e8a498d693") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:10.645332 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:10.645151    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:10.645332 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:10.645190    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls podName:6b2c3ca8-5f11-4fc2-a88c-70e8a498d693 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:12.645179173 +0000 UTC m=+73.653675305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4cllv" (UID: "6b2c3ca8-5f11-4fc2-a88c-70e8a498d693") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:10.920842 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:10.920766    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:19:10.922189 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:10.922162    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8dkfv" event={"ID":"bd509ebe-1474-4b42-88ae-96220ceecee8","Type":"ContainerStarted","Data":"d33fc0e3b90d0812eadb7b33051f4267d2b8174f232566da68fde947ebca7fd9"}
Sep 05 02:19:10.962463 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:10.962404    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8dkfv" podStartSLOduration=67.882787616 podStartE2EDuration="1m11.962364682s" totalImagesPullingTime="4.079577066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:10.961616594 +0000 UTC m=+71.970112752" watchObservedRunningTime="2026-09-05 02:19:10.962364682 +0000 UTC m=+71.970860821"
Sep 05 02:19:11.450862 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:11.450814    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:11.451031 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:11.450943    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:11.451031 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:11.450955    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-697d84c94b-6j7tj: secret "image-registry-tls" not found
Sep 05 02:19:11.451031 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:11.451001    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls podName:b1edddb2-de81-436c-bd1b-720e994f6e70 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:19.450988281 +0000 UTC m=+80.459484397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls") pod "image-registry-697d84c94b-6j7tj" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70") : secret "image-registry-tls" not found
Sep 05 02:19:11.753157 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:11.753054    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6s4p\" (UID: \"b0178d25-ade8-4d57-bacc-137fc466bbce\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:11.753513 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:11.753199    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 05 02:19:11.753513 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:11.753262    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert podName:b0178d25-ade8-4d57-bacc-137fc466bbce nodeName:}" failed. No retries permitted until 2026-09-05 02:19:15.753248374 +0000 UTC m=+76.761744490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6s4p" (UID: "b0178d25-ade8-4d57-bacc-137fc466bbce") : secret "networking-console-plugin-cert" not found
Sep 05 02:19:11.946124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:11.946094    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rfzkh_09e1b36b-9b93-4340-a648-fafc7e0b6f90/dns-node-resolver/0.log"
Sep 05 02:19:12.659431 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:12.659378    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:12.659431 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:12.659441    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:12.659722 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:12.659528    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:12.659722 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:12.659576    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:12.659722 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:12.659608    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm podName:6b2c3ca8-5f11-4fc2-a88c-70e8a498d693 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.659592765 +0000 UTC m=+77.668088881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4cllv" (UID: "6b2c3ca8-5f11-4fc2-a88c-70e8a498d693") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:12.659722 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:12.659640    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls podName:6b2c3ca8-5f11-4fc2-a88c-70e8a498d693 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.659622433 +0000 UTC m=+77.668118551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4cllv" (UID: "6b2c3ca8-5f11-4fc2-a88c-70e8a498d693") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:12.940800 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:12.940728    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8zqc9_06c658b0-5ad2-43c5-8057-633c5525d4ab/node-ca/0.log"
Sep 05 02:19:13.341368 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:13.341278    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rsph8_4f3f8d4a-ee71-418d-958e-3a92b0d250be/serve-healthcheck-canary/0.log"
Sep 05 02:19:13.837285 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:13.837249    2555 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:13.837464 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:13.837304    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:13.837665 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:13.837651    2555 scope.go:117] "RemoveContainer" containerID="922ec9d3f2feb90421b857683188a0b321af3a3a26fd6d67ecc1d00ab9dde578"
Sep 05 02:19:13.837809 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:13.837791    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vwlqs_openshift-console-operator(30f1f819-6e52-4eb7-b546-47f22a4d5f8b)\"" pod="openshift-console-operator/console-operator-85b454874b-vwlqs" podUID="30f1f819-6e52-4eb7-b546-47f22a4d5f8b"
Sep 05 02:19:15.786428 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:15.786396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6s4p\" (UID: \"b0178d25-ade8-4d57-bacc-137fc466bbce\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:15.786789 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:15.786556    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 05 02:19:15.786789 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:15.786625    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert podName:b0178d25-ade8-4d57-bacc-137fc466bbce nodeName:}" failed. No retries permitted until 2026-09-05 02:19:23.786608173 +0000 UTC m=+84.795104291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x6s4p" (UID: "b0178d25-ade8-4d57-bacc-137fc466bbce") : secret "networking-console-plugin-cert" not found
Sep 05 02:19:16.692581 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:16.692507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:16.692784 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:16.692639    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:16.692784 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:16.692660    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:16.692784 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:16.692705    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm podName:6b2c3ca8-5f11-4fc2-a88c-70e8a498d693 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:24.692687862 +0000 UTC m=+85.701183980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4cllv" (UID: "6b2c3ca8-5f11-4fc2-a88c-70e8a498d693") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:16.692940 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:16.692809    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:16.692940 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:16.692869    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls podName:6b2c3ca8-5f11-4fc2-a88c-70e8a498d693 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:24.692852005 +0000 UTC m=+85.701348137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4cllv" (UID: "6b2c3ca8-5f11-4fc2-a88c-70e8a498d693") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:19.514122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:19.514068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:19.516411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:19.516386    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls\") pod \"image-registry-697d84c94b-6j7tj\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") " pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:19.649640 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:19.649594    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:19.924204 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:19.924172    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-697d84c94b-6j7tj"]
Sep 05 02:19:19.926900 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:19.926871    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1edddb2_de81_436c_bd1b_720e994f6e70.slice/crio-365b62683c41ff13b10348a1de464d61d478bc533fc2fb7c043f61bbaac60a64 WatchSource:0}: Error finding container 365b62683c41ff13b10348a1de464d61d478bc533fc2fb7c043f61bbaac60a64: Status 404 returned error can't find the container with id 365b62683c41ff13b10348a1de464d61d478bc533fc2fb7c043f61bbaac60a64
Sep 05 02:19:19.949451 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:19.949425    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj" event={"ID":"b1edddb2-de81-436c-bd1b-720e994f6e70","Type":"ContainerStarted","Data":"365b62683c41ff13b10348a1de464d61d478bc533fc2fb7c043f61bbaac60a64"}
Sep 05 02:19:20.953961 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:20.953923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj" event={"ID":"b1edddb2-de81-436c-bd1b-720e994f6e70","Type":"ContainerStarted","Data":"4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136"}
Sep 05 02:19:20.954422 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:20.954042    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:20.978105 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:20.978053    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj" podStartSLOduration=17.978035571 podStartE2EDuration="17.978035571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:20.977353679 +0000 UTC m=+81.985849834" watchObservedRunningTime="2026-09-05 02:19:20.978035571 +0000 UTC m=+81.986531738"
Sep 05 02:19:23.845293 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:23.845157    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6s4p\" (UID: \"b0178d25-ade8-4d57-bacc-137fc466bbce\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:23.847612 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:23.847587    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0178d25-ade8-4d57-bacc-137fc466bbce-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x6s4p\" (UID: \"b0178d25-ade8-4d57-bacc-137fc466bbce\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:23.996036 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:23.995988    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p"
Sep 05 02:19:24.272189 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:24.272159    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-x6s4p"]
Sep 05 02:19:24.275067 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:24.275039    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0178d25_ade8_4d57_bacc_137fc466bbce.slice/crio-d094d7297006282e4fcb29e27a4407ad0cfc623d44d155c99065a6e147f3033f WatchSource:0}: Error finding container d094d7297006282e4fcb29e27a4407ad0cfc623d44d155c99065a6e147f3033f: Status 404 returned error can't find the container with id d094d7297006282e4fcb29e27a4407ad0cfc623d44d155c99065a6e147f3033f
Sep 05 02:19:24.754270 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:24.754236    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:24.754436 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:24.754307    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:24.754867 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:24.754847    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:24.756586 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:24.756561    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b2c3ca8-5f11-4fc2-a88c-70e8a498d693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cllv\" (UID: \"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693\") " pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:24.909374 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:24.909328    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4cllv"
Sep 05 02:19:24.967401 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:24.967372    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p" event={"ID":"b0178d25-ade8-4d57-bacc-137fc466bbce","Type":"ContainerStarted","Data":"d094d7297006282e4fcb29e27a4407ad0cfc623d44d155c99065a6e147f3033f"}
Sep 05 02:19:25.178603 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:25.178208    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4cllv"]
Sep 05 02:19:25.180845 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:25.180816    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2c3ca8_5f11_4fc2_a88c_70e8a498d693.slice/crio-24f5ebe2bfd9897facade001ee764173dae81eef2a28cbc413b30421a66bfa84 WatchSource:0}: Error finding container 24f5ebe2bfd9897facade001ee764173dae81eef2a28cbc413b30421a66bfa84: Status 404 returned error can't find the container with id 24f5ebe2bfd9897facade001ee764173dae81eef2a28cbc413b30421a66bfa84
Sep 05 02:19:25.972795 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:25.972755    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4cllv" event={"ID":"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693","Type":"ContainerStarted","Data":"eba56423c214720178c567ec3a428787e84c9d02b2818f1ad5be537d672536be"}
Sep 05 02:19:25.973189 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:25.972826    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4cllv" event={"ID":"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693","Type":"ContainerStarted","Data":"61f22ab666b4907dfa7fc07f164fe57da3ba63b36713f370f4b8b49f7790ec20"}
Sep 05 02:19:25.973189 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:25.972862    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4cllv" event={"ID":"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693","Type":"ContainerStarted","Data":"24f5ebe2bfd9897facade001ee764173dae81eef2a28cbc413b30421a66bfa84"}
Sep 05 02:19:26.978321 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:26.978288    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p" event={"ID":"b0178d25-ade8-4d57-bacc-137fc466bbce","Type":"ContainerStarted","Data":"61119288111ff07a5ae0a2174bdb1a7899f5a0f334ab5337f3fb01897301ae5e"}
Sep 05 02:19:26.993974 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:26.993916    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-x6s4p" podStartSLOduration=17.086542436 podStartE2EDuration="18.993899396s" totalImagesPullingTime="1.90735696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:26.993073184 +0000 UTC m=+88.001569342" watchObservedRunningTime="2026-09-05 02:19:26.993899396 +0000 UTC m=+88.002395564"
Sep 05 02:19:27.983948 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:27.983914    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4cllv" event={"ID":"6b2c3ca8-5f11-4fc2-a88c-70e8a498d693","Type":"ContainerStarted","Data":"03ee6afe1f6406173719b7d30dfa6f9cde7756453587accfc3c82813475d050c"}
Sep 05 02:19:28.002510 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:28.002457    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4cllv" podStartSLOduration=18.023460992 podStartE2EDuration="20.002443761s" totalImagesPullingTime="1.978982769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:19:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:28.002286641 +0000 UTC m=+89.010782813" watchObservedRunningTime="2026-09-05 02:19:28.002443761 +0000 UTC m=+89.010939899"
Sep 05 02:19:29.540102 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:29.540071    2555 scope.go:117] "RemoveContainer" containerID="922ec9d3f2feb90421b857683188a0b321af3a3a26fd6d67ecc1d00ab9dde578"
Sep 05 02:19:29.992021 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:29.991993    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:19:29.992188 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:29.992052    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vwlqs" event={"ID":"30f1f819-6e52-4eb7-b546-47f22a4d5f8b","Type":"ContainerStarted","Data":"01b6c03d6db927e776b555a7c0aab88e65ceb2df1c526f30bb359fd15be0efa7"}
Sep 05 02:19:29.992330 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:29.992307    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:30.012469 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.012413    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-vwlqs" podStartSLOduration=24.527616406 podStartE2EDuration="27.012393428s" totalImagesPullingTime="2.484777022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:30.011476622 +0000 UTC m=+91.019972781" watchObservedRunningTime="2026-09-05 02:19:30.012393428 +0000 UTC m=+91.020889569"
Sep 05 02:19:30.019639 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.019594    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-vwlqs"
Sep 05 02:19:30.750922 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.750890    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd"]
Sep 05 02:19:30.753991 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.753972    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd"
Sep 05 02:19:30.757233 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.757214    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 05 02:19:30.757467 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.757446    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 05 02:19:30.757886 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.757865    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 05 02:19:30.758398 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.758371    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jnzhl\""
Sep 05 02:19:30.759230 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.759208    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 05 02:19:30.781347 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.781311    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd"]
Sep 05 02:19:30.872010 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.871972    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-blnjb"]
Sep 05 02:19:30.874897 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.874876    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-blnjb"
Sep 05 02:19:30.885743 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.885700    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-49rcl\""
Sep 05 02:19:30.885861 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.885807    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 05 02:19:30.885861 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.885815    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 05 02:19:30.898010 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.897989    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-697d84c94b-6j7tj"]
Sep 05 02:19:30.901116 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.901091    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdx5k\" (UniqueName: \"kubernetes.io/projected/66d78e82-cd07-4929-9aab-aa9bec6070c5-kube-api-access-sdx5k\") pod \"managed-serviceaccount-addon-agent-769c78449b-pc5rd\" (UID: \"66d78e82-cd07-4929-9aab-aa9bec6070c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd"
Sep 05 02:19:30.901228 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.901137    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66d78e82-cd07-4929-9aab-aa9bec6070c5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-769c78449b-pc5rd\" (UID: \"66d78e82-cd07-4929-9aab-aa9bec6070c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd"
Sep 05 02:19:30.902328 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.902309    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-blnjb"]
Sep 05 02:19:30.998657 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:30.998625    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b55c9ff6-vs5m6"]
Sep 05 02:19:31.001695 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.001651    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.001695 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.001657    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdx5k\" (UniqueName: \"kubernetes.io/projected/66d78e82-cd07-4929-9aab-aa9bec6070c5-kube-api-access-sdx5k\") pod \"managed-serviceaccount-addon-agent-769c78449b-pc5rd\" (UID: \"66d78e82-cd07-4929-9aab-aa9bec6070c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd"
Sep 05 02:19:31.001695 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.001688    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66d78e82-cd07-4929-9aab-aa9bec6070c5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-769c78449b-pc5rd\" (UID: \"66d78e82-cd07-4929-9aab-aa9bec6070c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd"
Sep 05 02:19:31.001889 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.001724    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flstv\" (UniqueName: \"kubernetes.io/projected/c7611f52-960d-4175-a893-8c3efe168f2f-kube-api-access-flstv\") pod \"downloads-85d98c998f-blnjb\" (UID: \"c7611f52-960d-4175-a893-8c3efe168f2f\") " pod="openshift-console/downloads-85d98c998f-blnjb"
Sep 05 02:19:31.004153 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.004129    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66d78e82-cd07-4929-9aab-aa9bec6070c5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-769c78449b-pc5rd\" (UID: \"66d78e82-cd07-4929-9aab-aa9bec6070c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd"
Sep 05 02:19:31.030505 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.030485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdx5k\" (UniqueName: \"kubernetes.io/projected/66d78e82-cd07-4929-9aab-aa9bec6070c5-kube-api-access-sdx5k\") pod \"managed-serviceaccount-addon-agent-769c78449b-pc5rd\" (UID: \"66d78e82-cd07-4929-9aab-aa9bec6070c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd"
Sep 05 02:19:31.044683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.044657    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b55c9ff6-vs5m6"]
Sep 05 02:19:31.080249 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.080222    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd"
Sep 05 02:19:31.103054 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.103027    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59c7dd2d-013b-430c-bd53-e60d895f2365-installation-pull-secrets\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.103169 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.103085    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59c7dd2d-013b-430c-bd53-e60d895f2365-registry-certificates\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.103231 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.103206    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7zzt\" (UniqueName: \"kubernetes.io/projected/59c7dd2d-013b-430c-bd53-e60d895f2365-kube-api-access-r7zzt\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.103288 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.103254    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59c7dd2d-013b-430c-bd53-e60d895f2365-ca-trust-extracted\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.103348 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.103305    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59c7dd2d-013b-430c-bd53-e60d895f2365-trusted-ca\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.103421 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.103393    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59c7dd2d-013b-430c-bd53-e60d895f2365-image-registry-private-configuration\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.103474 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.103433    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59c7dd2d-013b-430c-bd53-e60d895f2365-registry-tls\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.103474 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.103451    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59c7dd2d-013b-430c-bd53-e60d895f2365-bound-sa-token\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.103566 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.103515    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flstv\" (UniqueName: \"kubernetes.io/projected/c7611f52-960d-4175-a893-8c3efe168f2f-kube-api-access-flstv\") pod \"downloads-85d98c998f-blnjb\" (UID: \"c7611f52-960d-4175-a893-8c3efe168f2f\") " pod="openshift-console/downloads-85d98c998f-blnjb"
Sep 05 02:19:31.117528 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.117503    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flstv\" (UniqueName: \"kubernetes.io/projected/c7611f52-960d-4175-a893-8c3efe168f2f-kube-api-access-flstv\") pod \"downloads-85d98c998f-blnjb\" (UID: \"c7611f52-960d-4175-a893-8c3efe168f2f\") " pod="openshift-console/downloads-85d98c998f-blnjb"
Sep 05 02:19:31.182942 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.182905    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-blnjb"
Sep 05 02:19:31.204221 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.204187    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59c7dd2d-013b-430c-bd53-e60d895f2365-ca-trust-extracted\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.204221 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.204222    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59c7dd2d-013b-430c-bd53-e60d895f2365-trusted-ca\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.204364 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.204243    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59c7dd2d-013b-430c-bd53-e60d895f2365-image-registry-private-configuration\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.204578 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.204520    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59c7dd2d-013b-430c-bd53-e60d895f2365-registry-tls\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.204660 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.204605    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59c7dd2d-013b-430c-bd53-e60d895f2365-bound-sa-token\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.204716 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.204689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59c7dd2d-013b-430c-bd53-e60d895f2365-installation-pull-secrets\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.204764 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.204722    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59c7dd2d-013b-430c-bd53-e60d895f2365-registry-certificates\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.204815 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.204794    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59c7dd2d-013b-430c-bd53-e60d895f2365-ca-trust-extracted\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.204869 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.204802    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zzt\" (UniqueName: \"kubernetes.io/projected/59c7dd2d-013b-430c-bd53-e60d895f2365-kube-api-access-r7zzt\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.205321 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.205300    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59c7dd2d-013b-430c-bd53-e60d895f2365-trusted-ca\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.205634 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.205606    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59c7dd2d-013b-430c-bd53-e60d895f2365-registry-certificates\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.207105 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.207079    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59c7dd2d-013b-430c-bd53-e60d895f2365-installation-pull-secrets\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.207200 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.207183    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59c7dd2d-013b-430c-bd53-e60d895f2365-registry-tls\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.207259 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.207244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59c7dd2d-013b-430c-bd53-e60d895f2365-image-registry-private-configuration\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.229758 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.229730    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59c7dd2d-013b-430c-bd53-e60d895f2365-bound-sa-token\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.229871 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.229852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zzt\" (UniqueName: \"kubernetes.io/projected/59c7dd2d-013b-430c-bd53-e60d895f2365-kube-api-access-r7zzt\") pod \"image-registry-7b55c9ff6-vs5m6\" (UID: \"59c7dd2d-013b-430c-bd53-e60d895f2365\") " pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.314899 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.314803    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:31.380239 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.380211    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd"]
Sep 05 02:19:31.381376 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:31.381321    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66d78e82_cd07_4929_9aab_aa9bec6070c5.slice/crio-561aba611bf67088f610dc2956f84cbc2a2ea9f01eed53a7ed4bbe0d6a00a1b2 WatchSource:0}: Error finding container 561aba611bf67088f610dc2956f84cbc2a2ea9f01eed53a7ed4bbe0d6a00a1b2: Status 404 returned error can't find the container with id 561aba611bf67088f610dc2956f84cbc2a2ea9f01eed53a7ed4bbe0d6a00a1b2
Sep 05 02:19:31.456568 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.456514    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-blnjb"]
Sep 05 02:19:31.459729 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:31.459705    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7611f52_960d_4175_a893_8c3efe168f2f.slice/crio-9a7bc894c766608189ceb2d7ea15d94ef6a41887860c5d321cf73a55d60f8bc5 WatchSource:0}: Error finding container 9a7bc894c766608189ceb2d7ea15d94ef6a41887860c5d321cf73a55d60f8bc5: Status 404 returned error can't find the container with id 9a7bc894c766608189ceb2d7ea15d94ef6a41887860c5d321cf73a55d60f8bc5
Sep 05 02:19:31.607473 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:31.607437    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c7dd2d_013b_430c_bd53_e60d895f2365.slice/crio-9b57545a0dda61dec65290c1b5f3692d6c678a905c60957b0dac0d0ed31f9e67 WatchSource:0}: Error finding container 9b57545a0dda61dec65290c1b5f3692d6c678a905c60957b0dac0d0ed31f9e67: Status 404 returned error can't find the container with id 9b57545a0dda61dec65290c1b5f3692d6c678a905c60957b0dac0d0ed31f9e67
Sep 05 02:19:31.611767 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:31.611742    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b55c9ff6-vs5m6"]
Sep 05 02:19:32.001978 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:32.001942    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6" event={"ID":"59c7dd2d-013b-430c-bd53-e60d895f2365","Type":"ContainerStarted","Data":"ea1b4d6798be4ce3ef2fc155a66af62bac208ff901f991fe9a8702490e00658d"}
Sep 05 02:19:32.002454 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:32.002000    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6" event={"ID":"59c7dd2d-013b-430c-bd53-e60d895f2365","Type":"ContainerStarted","Data":"9b57545a0dda61dec65290c1b5f3692d6c678a905c60957b0dac0d0ed31f9e67"}
Sep 05 02:19:32.002577 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:32.002556    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:32.003767 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:32.003739    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-blnjb" event={"ID":"c7611f52-960d-4175-a893-8c3efe168f2f","Type":"ContainerStarted","Data":"9a7bc894c766608189ceb2d7ea15d94ef6a41887860c5d321cf73a55d60f8bc5"}
Sep 05 02:19:32.005125 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:32.005074    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd" event={"ID":"66d78e82-cd07-4929-9aab-aa9bec6070c5","Type":"ContainerStarted","Data":"561aba611bf67088f610dc2956f84cbc2a2ea9f01eed53a7ed4bbe0d6a00a1b2"}
Sep 05 02:19:32.051805 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:32.051754    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6" podStartSLOduration=2.051736792 podStartE2EDuration="2.051736792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:32.050949865 +0000 UTC m=+93.059446027" watchObservedRunningTime="2026-09-05 02:19:32.051736792 +0000 UTC m=+93.060232930"
Sep 05 02:19:35.020162 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:35.020121    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd" event={"ID":"66d78e82-cd07-4929-9aab-aa9bec6070c5","Type":"ContainerStarted","Data":"c56e45feaca8965718c606ae027e686a4f13b63a74cc48546e2bd82d5de9d22e"}
Sep 05 02:19:35.037197 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:35.037134    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769c78449b-pc5rd" podStartSLOduration=1.589449964 podStartE2EDuration="5.03711585s" totalImagesPullingTime="3.447665886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:35.036199953 +0000 UTC m=+96.044696118" watchObservedRunningTime="2026-09-05 02:19:35.03711585 +0000 UTC m=+96.045611988"
Sep 05 02:19:37.260317 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:37.260276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:19:37.262807 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:37.262776    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0769ef0-90d9-43ee-ade4-5aae6141a935-metrics-tls\") pod \"dns-default-9l9d9\" (UID: \"f0769ef0-90d9-43ee-ade4-5aae6141a935\") " pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:19:37.538923 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:37.538842    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2swhn\""
Sep 05 02:19:37.547522 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:37.547496    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:19:37.832467 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:37.832441    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9l9d9"]
Sep 05 02:19:37.835092 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:37.835067    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0769ef0_90d9_43ee_ade4_5aae6141a935.slice/crio-9318b20660355bc1d09e2896d849b486b5500c2d86ab28e3e78b0e9dd33e30fb WatchSource:0}: Error finding container 9318b20660355bc1d09e2896d849b486b5500c2d86ab28e3e78b0e9dd33e30fb: Status 404 returned error can't find the container with id 9318b20660355bc1d09e2896d849b486b5500c2d86ab28e3e78b0e9dd33e30fb
Sep 05 02:19:38.037410 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:38.037366    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9l9d9" event={"ID":"f0769ef0-90d9-43ee-ade4-5aae6141a935","Type":"ContainerStarted","Data":"9318b20660355bc1d09e2896d849b486b5500c2d86ab28e3e78b0e9dd33e30fb"}
Sep 05 02:19:39.919662 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:39.919635    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8t685"
Sep 05 02:19:40.050937 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:40.050898    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9l9d9" event={"ID":"f0769ef0-90d9-43ee-ade4-5aae6141a935","Type":"ContainerStarted","Data":"bcbf0db78e07845f777c5d5c8f962c82ac4c99681dfb1c99a879d9564b4d1068"}
Sep 05 02:19:40.051112 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:40.050965    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9l9d9" event={"ID":"f0769ef0-90d9-43ee-ade4-5aae6141a935","Type":"ContainerStarted","Data":"40d622b7a8b0e7717902fd90984c08b82a1af91f871e4509a5161ec26921cb5b"}
Sep 05 02:19:40.051112 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:40.051055    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:19:40.105837 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:40.105770    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9l9d9" podStartSLOduration=65.191046717 podStartE2EDuration="1m7.105745921s" totalImagesPullingTime="1.914699204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:40.105198597 +0000 UTC m=+101.113694749" watchObservedRunningTime="2026-09-05 02:19:40.105745921 +0000 UTC m=+101.114242061"
Sep 05 02:19:40.903989 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:40.903964    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:41.071993 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.071952    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-cvshl"]
Sep 05 02:19:41.090898 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.090833    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-cvshl"]
Sep 05 02:19:41.091070 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.090923    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.095710 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.095687    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 05 02:19:41.095869 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.095850    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 05 02:19:41.095949 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.095926    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wq2z9\""
Sep 05 02:19:41.096010 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.095971    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 05 02:19:41.096010 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.095992    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 05 02:19:41.096849 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.096830    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 05 02:19:41.194427 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.194340    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cfb41ee-1e4c-4673-87ff-58216a84edd8-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.194427 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.194408    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnsw4\" (UniqueName: \"kubernetes.io/projected/0cfb41ee-1e4c-4673-87ff-58216a84edd8-kube-api-access-cnsw4\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.194688 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.194456    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0cfb41ee-1e4c-4673-87ff-58216a84edd8-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.194688 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.194520    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cfb41ee-1e4c-4673-87ff-58216a84edd8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.294919 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.294886    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0cfb41ee-1e4c-4673-87ff-58216a84edd8-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.295107 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.294951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cfb41ee-1e4c-4673-87ff-58216a84edd8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.295107 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.295018    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cfb41ee-1e4c-4673-87ff-58216a84edd8-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.295107 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.295062    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsw4\" (UniqueName: \"kubernetes.io/projected/0cfb41ee-1e4c-4673-87ff-58216a84edd8-kube-api-access-cnsw4\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.295847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.295818    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cfb41ee-1e4c-4673-87ff-58216a84edd8-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.297829 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.297802    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cfb41ee-1e4c-4673-87ff-58216a84edd8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.297958 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.297804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0cfb41ee-1e4c-4673-87ff-58216a84edd8-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.308986 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.308960    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnsw4\" (UniqueName: \"kubernetes.io/projected/0cfb41ee-1e4c-4673-87ff-58216a84edd8-kube-api-access-cnsw4\") pod \"prometheus-operator-6f4946db56-cvshl\" (UID: \"0cfb41ee-1e4c-4673-87ff-58216a84edd8\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:41.402809 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:41.402774    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl"
Sep 05 02:19:48.083416 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:48.083379    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-blnjb" event={"ID":"c7611f52-960d-4175-a893-8c3efe168f2f","Type":"ContainerStarted","Data":"b635e7d4bb70bcf9a7c7f77af112cdc199300bfa44ea489ee42c09f4bdacb504"}
Sep 05 02:19:48.083949 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:48.083573    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-blnjb"
Sep 05 02:19:48.085085 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:48.085053    2555 patch_prober.go:28] interesting pod/downloads-85d98c998f-blnjb container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" start-of-body=
Sep 05 02:19:48.085196 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:48.085106    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-blnjb" podUID="c7611f52-960d-4175-a893-8c3efe168f2f" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused"
Sep 05 02:19:48.107936 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:48.107844    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-blnjb" podStartSLOduration=1.654891597 podStartE2EDuration="18.107824256s" totalImagesPullingTime="16.452932659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:48.10627846 +0000 UTC m=+109.114774622" watchObservedRunningTime="2026-09-05 02:19:48.107824256 +0000 UTC m=+109.116320394"
Sep 05 02:19:48.114078 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:48.114047    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-cvshl"]
Sep 05 02:19:48.117704 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:48.117670    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cfb41ee_1e4c_4673_87ff_58216a84edd8.slice/crio-44bbbb45f412ff90b5d6adebdf8c91c64266479bdc9fd9958664e15a156757ac WatchSource:0}: Error finding container 44bbbb45f412ff90b5d6adebdf8c91c64266479bdc9fd9958664e15a156757ac: Status 404 returned error can't find the container with id 44bbbb45f412ff90b5d6adebdf8c91c64266479bdc9fd9958664e15a156757ac
Sep 05 02:19:49.090407 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:49.090327    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl" event={"ID":"0cfb41ee-1e4c-4673-87ff-58216a84edd8","Type":"ContainerStarted","Data":"44bbbb45f412ff90b5d6adebdf8c91c64266479bdc9fd9958664e15a156757ac"}
Sep 05 02:19:49.104949 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:49.104916    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-blnjb"
Sep 05 02:19:50.058095 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:50.058061    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9l9d9"
Sep 05 02:19:50.098823 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:50.098773    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl" event={"ID":"0cfb41ee-1e4c-4673-87ff-58216a84edd8","Type":"ContainerStarted","Data":"42b6717282724da809d6373fa5417db12b1612dff00b79b30c4cefde8379ca7b"}
Sep 05 02:19:50.099280 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:50.098844    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl" event={"ID":"0cfb41ee-1e4c-4673-87ff-58216a84edd8","Type":"ContainerStarted","Data":"129681847c6b7b5f09b8a98dc158943ff668f10c210351ef83824debf79264e4"}
Sep 05 02:19:50.117906 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:50.117846    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-cvshl" podStartSLOduration=7.945867958 podStartE2EDuration="9.117824016s" totalImagesPullingTime="1.171956058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:50.11676162 +0000 UTC m=+111.125257761" watchObservedRunningTime="2026-09-05 02:19:50.117824016 +0000 UTC m=+111.126320159"
Sep 05 02:19:52.417100 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.417060    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"]
Sep 05 02:19:52.439476 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.439443    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7hxvj"]
Sep 05 02:19:52.439666 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.439595    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.443316 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.443290    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 05 02:19:52.443442 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.443290    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 05 02:19:52.443683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.443660    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 05 02:19:52.443795 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.443775    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bqzdt\""
Sep 05 02:19:52.457254 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.457233    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"]
Sep 05 02:19:52.457385 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.457361    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.459862 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.459837    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dlqp5\""
Sep 05 02:19:52.459959 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.459911    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 05 02:19:52.459959 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.459933    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 05 02:19:52.459959 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.459911    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 05 02:19:52.487968 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.487935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.488103 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.487982    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-textfile\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.488103 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.488103 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488045    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbpl\" (UniqueName: \"kubernetes.io/projected/4259cc42-4148-48c0-917d-b3e88155d4a2-kube-api-access-ljbpl\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.488266 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488105    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.488266 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488145    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.488266 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488203    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3dc1bfa3-ce6a-4794-911b-959cadd385a4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.488397 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488255    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-wtmp\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.488397 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488293    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4259cc42-4148-48c0-917d-b3e88155d4a2-metrics-client-ca\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.488397 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488323    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4259cc42-4148-48c0-917d-b3e88155d4a2-sys\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.488397 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488349    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-tls\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.488397 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488375    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v444\" (UniqueName: \"kubernetes.io/projected/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-api-access-8v444\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.488657 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488404    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.488657 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488447    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3dc1bfa3-ce6a-4794-911b-959cadd385a4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.488657 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.488485    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4259cc42-4148-48c0-917d-b3e88155d4a2-root\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.589675 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.589634    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3dc1bfa3-ce6a-4794-911b-959cadd385a4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.589675 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.589677    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4259cc42-4148-48c0-917d-b3e88155d4a2-root\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.589918 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.589730    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.589918 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.589758    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-textfile\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.589918 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.589792    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.589918 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.589838    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4259cc42-4148-48c0-917d-b3e88155d4a2-root\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.590133 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.589980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbpl\" (UniqueName: \"kubernetes.io/projected/4259cc42-4148-48c0-917d-b3e88155d4a2-kube-api-access-ljbpl\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.590133 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590046    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.590133 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590089    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.590133 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590126    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3dc1bfa3-ce6a-4794-911b-959cadd385a4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.590349 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590140    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-textfile\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.590349 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590158    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-wtmp\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.590349 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590183    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4259cc42-4148-48c0-917d-b3e88155d4a2-metrics-client-ca\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.590349 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590209    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4259cc42-4148-48c0-917d-b3e88155d4a2-sys\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.590349 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590235    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-tls\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.590349 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590260    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v444\" (UniqueName: \"kubernetes.io/projected/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-api-access-8v444\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.590349 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590291    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.590843 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:52.590818    2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 05 02:19:52.590927 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590864    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.590927 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590864    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3dc1bfa3-ce6a-4794-911b-959cadd385a4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.590927 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:52.590893    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-tls podName:4259cc42-4148-48c0-917d-b3e88155d4a2 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:53.090872273 +0000 UTC m=+114.099368389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-tls") pod "node-exporter-7hxvj" (UID: "4259cc42-4148-48c0-917d-b3e88155d4a2") : secret "node-exporter-tls" not found
Sep 05 02:19:52.591108 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.591073    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4259cc42-4148-48c0-917d-b3e88155d4a2-sys\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.591108 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.590124    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3dc1bfa3-ce6a-4794-911b-959cadd385a4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.591244 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.591221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-wtmp\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.591311 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.591259    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4259cc42-4148-48c0-917d-b3e88155d4a2-metrics-client-ca\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.591627 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.591598    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.592675 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.592648    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.592813 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.592767    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.593350 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.593327    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.598426 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.598403    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbpl\" (UniqueName: \"kubernetes.io/projected/4259cc42-4148-48c0-917d-b3e88155d4a2-kube-api-access-ljbpl\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:52.601406 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.601386    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v444\" (UniqueName: \"kubernetes.io/projected/3dc1bfa3-ce6a-4794-911b-959cadd385a4-kube-api-access-8v444\") pod \"kube-state-metrics-74cb57df7-nt84q\" (UID: \"3dc1bfa3-ce6a-4794-911b-959cadd385a4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:52.752200 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:52.752117    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"
Sep 05 02:19:53.065856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.065723    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-nt84q"]
Sep 05 02:19:53.069815 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:53.069781    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc1bfa3_ce6a_4794_911b_959cadd385a4.slice/crio-514309336b197e5c7bdcd207e1427c1e0c6ca59c32aa87174682912e19a55e1e WatchSource:0}: Error finding container 514309336b197e5c7bdcd207e1427c1e0c6ca59c32aa87174682912e19a55e1e: Status 404 returned error can't find the container with id 514309336b197e5c7bdcd207e1427c1e0c6ca59c32aa87174682912e19a55e1e
Sep 05 02:19:53.095806 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.095774    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-tls\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:53.098378 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.098351    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4259cc42-4148-48c0-917d-b3e88155d4a2-node-exporter-tls\") pod \"node-exporter-7hxvj\" (UID: \"4259cc42-4148-48c0-917d-b3e88155d4a2\") " pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:53.111928 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.111891    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q" event={"ID":"3dc1bfa3-ce6a-4794-911b-959cadd385a4","Type":"ContainerStarted","Data":"514309336b197e5c7bdcd207e1427c1e0c6ca59c32aa87174682912e19a55e1e"}
Sep 05 02:19:53.368602 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.368504    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7hxvj"
Sep 05 02:19:53.407658 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:53.407622    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4259cc42_4148_48c0_917d_b3e88155d4a2.slice/crio-48ea8463bc38d6a5c9998e7f52fbc85b7a085e37002b14db918d0b1647737e47 WatchSource:0}: Error finding container 48ea8463bc38d6a5c9998e7f52fbc85b7a085e37002b14db918d0b1647737e47: Status 404 returned error can't find the container with id 48ea8463bc38d6a5c9998e7f52fbc85b7a085e37002b14db918d0b1647737e47
Sep 05 02:19:53.586495 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.586455    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:19:53.597615 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.597586    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.601488 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.601249    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 05 02:19:53.601488 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.601275    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 05 02:19:53.606511 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.604410    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mg4dt\""
Sep 05 02:19:53.606511 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.604689    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 05 02:19:53.606511 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.605492    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 05 02:19:53.606511 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.605758    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 05 02:19:53.606907 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.606887    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 05 02:19:53.607113 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.607095    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 05 02:19:53.607345 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.607327    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 05 02:19:53.607721 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.607698    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 05 02:19:53.615384 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.615307    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:19:53.702450 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.702415    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-volume\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.703158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.702470    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.703158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.702506    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.703158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.702554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.703158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.702585    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.703158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.702633    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.703158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.702698    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-web-config\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.703158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.702752    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-out\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.703158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.702818    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.703158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.702939    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.703158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.702987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.803519 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803472    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.803717 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803569    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.803717 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.803717 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-volume\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.803717 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803697    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.803937 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803728    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.803937 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803759    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.803937 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.803937 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803889    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j72d\" (UniqueName: \"kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-kube-api-access-6j72d\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.803937 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803918    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.804143 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803945    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.804143 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.803974    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-web-config\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.804143 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.804005    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-out\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.804836 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.804806    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.805891 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.805866    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.810767 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.810714    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.812098 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.812071    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-web-config\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.812870 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.812817    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.813175 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.812981    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-volume\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.813175 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.812988    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.813338 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.813241    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.813637 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.813502    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.815044 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.815019    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.815476 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.815451    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-out\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.905912 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.904686    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j72d\" (UniqueName: \"kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-kube-api-access-6j72d\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.905912 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.904756    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.906462 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.906437    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.921442 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.921312    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j72d\" (UniqueName: \"kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-kube-api-access-6j72d\") pod \"alertmanager-main-0\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:53.923221 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:53.923167    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:54.024451 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:54.024198    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b55c9ff6-vs5m6"
Sep 05 02:19:54.118939 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:54.118881    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hxvj" event={"ID":"4259cc42-4148-48c0-917d-b3e88155d4a2","Type":"ContainerStarted","Data":"48ea8463bc38d6a5c9998e7f52fbc85b7a085e37002b14db918d0b1647737e47"}
Sep 05 02:19:54.259157 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:54.259127    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:19:54.593246 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:19:54.593171    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b5fc89_3dfe_4b5d_9787_cceb72925fb7.slice/crio-8fe06ae0ab8fe70b2d15743ae794f82b6bcbb88ca37f5b325f32d5e10691fcf4 WatchSource:0}: Error finding container 8fe06ae0ab8fe70b2d15743ae794f82b6bcbb88ca37f5b325f32d5e10691fcf4: Status 404 returned error can't find the container with id 8fe06ae0ab8fe70b2d15743ae794f82b6bcbb88ca37f5b325f32d5e10691fcf4
Sep 05 02:19:55.124620 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:55.124570    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerStarted","Data":"8fe06ae0ab8fe70b2d15743ae794f82b6bcbb88ca37f5b325f32d5e10691fcf4"}
Sep 05 02:19:55.917906 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:55.917657    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj" podUID="b1edddb2-de81-436c-bd1b-720e994f6e70" containerName="registry" containerID="cri-o://4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136" gracePeriod=30
Sep 05 02:19:56.132054 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.132009    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q" event={"ID":"3dc1bfa3-ce6a-4794-911b-959cadd385a4","Type":"ContainerStarted","Data":"3e646c31924b2458317bcf54e2b7cfe80659d48ea7c285249b6e3ec49316cead"}
Sep 05 02:19:56.132226 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.132082    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q" event={"ID":"3dc1bfa3-ce6a-4794-911b-959cadd385a4","Type":"ContainerStarted","Data":"0beee23e26994edf0d0f69f49cfaf2f2ec6e8c4a222bca22d6c16379e050ac24"}
Sep 05 02:19:56.134052 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.134019    2555 generic.go:356] "Generic (PLEG): container finished" podID="4259cc42-4148-48c0-917d-b3e88155d4a2" containerID="0240b2fc11130ca4cb79d11b7606b6ab13546e9bb345ba8845da89b600f68afd" exitCode=0
Sep 05 02:19:56.134169 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.134095    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hxvj" event={"ID":"4259cc42-4148-48c0-917d-b3e88155d4a2","Type":"ContainerDied","Data":"0240b2fc11130ca4cb79d11b7606b6ab13546e9bb345ba8845da89b600f68afd"}
Sep 05 02:19:56.514054 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.514018    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:56.529824 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.528208    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-certificates\") pod \"b1edddb2-de81-436c-bd1b-720e994f6e70\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") "
Sep 05 02:19:56.529824 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.528248    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6mnp\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-kube-api-access-n6mnp\") pod \"b1edddb2-de81-436c-bd1b-720e994f6e70\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") "
Sep 05 02:19:56.529824 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.528318    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls\") pod \"b1edddb2-de81-436c-bd1b-720e994f6e70\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") "
Sep 05 02:19:56.529824 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.528356    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-bound-sa-token\") pod \"b1edddb2-de81-436c-bd1b-720e994f6e70\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") "
Sep 05 02:19:56.529824 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.528389    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-image-registry-private-configuration\") pod \"b1edddb2-de81-436c-bd1b-720e994f6e70\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") "
Sep 05 02:19:56.529824 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.528428    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1edddb2-de81-436c-bd1b-720e994f6e70-ca-trust-extracted\") pod \"b1edddb2-de81-436c-bd1b-720e994f6e70\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") "
Sep 05 02:19:56.529824 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.528458    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-trusted-ca\") pod \"b1edddb2-de81-436c-bd1b-720e994f6e70\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") "
Sep 05 02:19:56.529824 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.528485    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-installation-pull-secrets\") pod \"b1edddb2-de81-436c-bd1b-720e994f6e70\" (UID: \"b1edddb2-de81-436c-bd1b-720e994f6e70\") "
Sep 05 02:19:56.529824 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.528698    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b1edddb2-de81-436c-bd1b-720e994f6e70" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:19:56.529824 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.529203    2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-certificates\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:19:56.531205 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.531134    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b1edddb2-de81-436c-bd1b-720e994f6e70" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:19:56.531306 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.531209    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b1edddb2-de81-436c-bd1b-720e994f6e70" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:19:56.531385 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.531362    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-kube-api-access-n6mnp" (OuterVolumeSpecName: "kube-api-access-n6mnp") pod "b1edddb2-de81-436c-bd1b-720e994f6e70" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70"). InnerVolumeSpecName "kube-api-access-n6mnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:19:56.537924 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.531469    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b1edddb2-de81-436c-bd1b-720e994f6e70" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:19:56.537924 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.532180    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b1edddb2-de81-436c-bd1b-720e994f6e70" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:19:56.538368 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.538251    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b1edddb2-de81-436c-bd1b-720e994f6e70" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:19:56.545870 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.545817    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1edddb2-de81-436c-bd1b-720e994f6e70-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b1edddb2-de81-436c-bd1b-720e994f6e70" (UID: "b1edddb2-de81-436c-bd1b-720e994f6e70"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:19:56.629872 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.629837    2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-registry-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:19:56.629872 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.629869    2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-bound-sa-token\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:19:56.629872 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.629879    2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-image-registry-private-configuration\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:19:56.630094 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.629894    2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1edddb2-de81-436c-bd1b-720e994f6e70-ca-trust-extracted\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:19:56.630094 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.629908    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1edddb2-de81-436c-bd1b-720e994f6e70-trusted-ca\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:19:56.630094 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.629919    2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1edddb2-de81-436c-bd1b-720e994f6e70-installation-pull-secrets\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:19:56.630094 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:56.629933    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6mnp\" (UniqueName: \"kubernetes.io/projected/b1edddb2-de81-436c-bd1b-720e994f6e70-kube-api-access-n6mnp\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:19:57.141424 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.141375    2555 generic.go:356] "Generic (PLEG): container finished" podID="b1edddb2-de81-436c-bd1b-720e994f6e70" containerID="4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136" exitCode=0
Sep 05 02:19:57.141892 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.141436    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj"
Sep 05 02:19:57.141892 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.141497    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj" event={"ID":"b1edddb2-de81-436c-bd1b-720e994f6e70","Type":"ContainerDied","Data":"4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136"}
Sep 05 02:19:57.141892 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.141566    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-697d84c94b-6j7tj" event={"ID":"b1edddb2-de81-436c-bd1b-720e994f6e70","Type":"ContainerDied","Data":"365b62683c41ff13b10348a1de464d61d478bc533fc2fb7c043f61bbaac60a64"}
Sep 05 02:19:57.141892 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.141594    2555 scope.go:117] "RemoveContainer" containerID="4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136"
Sep 05 02:19:57.145259 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.145189    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q" event={"ID":"3dc1bfa3-ce6a-4794-911b-959cadd385a4","Type":"ContainerStarted","Data":"415b1433fc8bee7330848d6cd9eb0e3625298233d8841ec6bd8de9a1eee36a96"}
Sep 05 02:19:57.150730 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.150701    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hxvj" event={"ID":"4259cc42-4148-48c0-917d-b3e88155d4a2","Type":"ContainerStarted","Data":"344d117781118f6d49cdc99d5c6bc34135f9e4c4083ba715f3a447d8cf27f77f"}
Sep 05 02:19:57.150826 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.150760    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hxvj" event={"ID":"4259cc42-4148-48c0-917d-b3e88155d4a2","Type":"ContainerStarted","Data":"29766913c4b0626e2b968b314415068a906281c00dc245cd2abe7f34487c2e27"}
Sep 05 02:19:57.153386 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.153366    2555 scope.go:117] "RemoveContainer" containerID="4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136"
Sep 05 02:19:57.153973 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:19:57.153937    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136\": container with ID starting with 4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136 not found: ID does not exist" containerID="4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136"
Sep 05 02:19:57.154069 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.153986    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136"} err="failed to get container status \"4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136\": rpc error: code = NotFound desc = could not find container \"4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136\": container with ID starting with 4ca76dd099203f71aea7feae6b57c909b9980265b2239971baf3f8f1e20f0136 not found: ID does not exist"
Sep 05 02:19:57.154393 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.154367    2555 generic.go:356] "Generic (PLEG): container finished" podID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerID="11fdf70555c2021766171ca4a0e148bb7720405502cb5aeaa86780e866d295d3" exitCode=0
Sep 05 02:19:57.154490 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.154468    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerDied","Data":"11fdf70555c2021766171ca4a0e148bb7720405502cb5aeaa86780e866d295d3"}
Sep 05 02:19:57.176325 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.176268    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nt84q" podStartSLOduration=2.834698697 podStartE2EDuration="5.176245682s" totalImagesPullingTime="2.341546985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:57.174764052 +0000 UTC m=+118.183260212" watchObservedRunningTime="2026-09-05 02:19:57.176245682 +0000 UTC m=+118.184741824"
Sep 05 02:19:57.201469 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.201411    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7hxvj" podStartSLOduration=3.0647582 podStartE2EDuration="5.201395108s" totalImagesPullingTime="2.00642704s" totalInitContainerRuntime="130.209868ms" isStatefulPod=true podCreationTimestamp="2026-09-05 02:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:57.200773698 +0000 UTC m=+118.209269870" watchObservedRunningTime="2026-09-05 02:19:57.201395108 +0000 UTC m=+118.209891247"
Sep 05 02:19:57.223669 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.223631    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-697d84c94b-6j7tj"]
Sep 05 02:19:57.237222 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.237194    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-697d84c94b-6j7tj"]
Sep 05 02:19:57.543203 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:57.543136    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1edddb2-de81-436c-bd1b-720e994f6e70" path="/var/lib/kubelet/pods/b1edddb2-de81-436c-bd1b-720e994f6e70/volumes"
Sep 05 02:19:59.170883 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:59.170853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerStarted","Data":"c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1"}
Sep 05 02:19:59.171348 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:19:59.170915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerStarted","Data":"ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40"}
Sep 05 02:20:00.177606 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:00.177571    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerStarted","Data":"2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608"}
Sep 05 02:20:00.178045 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:00.177641    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerStarted","Data":"84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01"}
Sep 05 02:20:00.178045 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:00.177667    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerStarted","Data":"405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05"}
Sep 05 02:20:02.188490 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:02.188404    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerStarted","Data":"8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148"}
Sep 05 02:20:02.216908 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:02.216856    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.095484873 podStartE2EDuration="9.216841102s" totalImagesPullingTime="6.102697131s" totalInitContainerRuntime="18.659098ms" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:02.214844655 +0000 UTC m=+123.223340853" watchObservedRunningTime="2026-09-05 02:20:02.216841102 +0000 UTC m=+123.225337240"
Sep 05 02:20:09.340715 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:09.340674    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:20:09.342949 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:09.342927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd23bdc-d907-4b00-95c5-24828e92da98-metrics-certs\") pod \"network-metrics-daemon-85xds\" (UID: \"2dd23bdc-d907-4b00-95c5-24828e92da98\") " pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:20:09.554207 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:09.554172    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ptsx2\""
Sep 05 02:20:09.561606 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:09.561577    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85xds"
Sep 05 02:20:09.862211 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:09.862126    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-85xds"]
Sep 05 02:20:09.864799 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:20:09.864769    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd23bdc_d907_4b00_95c5_24828e92da98.slice/crio-9395a80871243d15bd95d3fcbfef77affdf66388562f1f463648c9d4154413e1 WatchSource:0}: Error finding container 9395a80871243d15bd95d3fcbfef77affdf66388562f1f463648c9d4154413e1: Status 404 returned error can't find the container with id 9395a80871243d15bd95d3fcbfef77affdf66388562f1f463648c9d4154413e1
Sep 05 02:20:10.222247 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:10.222213    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85xds" event={"ID":"2dd23bdc-d907-4b00-95c5-24828e92da98","Type":"ContainerStarted","Data":"9395a80871243d15bd95d3fcbfef77affdf66388562f1f463648c9d4154413e1"}
Sep 05 02:20:12.232178 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:12.232140    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85xds" event={"ID":"2dd23bdc-d907-4b00-95c5-24828e92da98","Type":"ContainerStarted","Data":"69a28efcef27aadd94106fede9374964247909868bec3b094839dbf443ea3493"}
Sep 05 02:20:12.232581 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:12.232196    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85xds" event={"ID":"2dd23bdc-d907-4b00-95c5-24828e92da98","Type":"ContainerStarted","Data":"6652ee9acd147f8561a392d914100873740ade9c9635ea81d388c9e783488e5f"}
Sep 05 02:20:12.251401 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:20:12.251347    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-85xds" podStartSLOduration=131.865348466 podStartE2EDuration="2m13.251329888s" totalImagesPullingTime="1.385981422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:12.249688353 +0000 UTC m=+133.258184503" watchObservedRunningTime="2026-09-05 02:20:12.251329888 +0000 UTC m=+133.259826079"
Sep 05 02:21:12.745258 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:12.745154    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:21:12.745804 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:12.745559    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="alertmanager" containerID="cri-o://ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40" gracePeriod=120
Sep 05 02:21:12.745804 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:12.745615    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy" containerID="cri-o://84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01" gracePeriod=120
Sep 05 02:21:12.745804 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:12.745664    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy-metric" containerID="cri-o://2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608" gracePeriod=120
Sep 05 02:21:12.745804 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:12.745676    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="config-reloader" containerID="cri-o://c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1" gracePeriod=120
Sep 05 02:21:12.745804 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:12.745622    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy-web" containerID="cri-o://405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05" gracePeriod=120
Sep 05 02:21:12.745804 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:12.745620    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="prom-label-proxy" containerID="cri-o://8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148" gracePeriod=120
Sep 05 02:21:13.468023 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:13.467986    2555 generic.go:356] "Generic (PLEG): container finished" podID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerID="8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148" exitCode=0
Sep 05 02:21:13.468023 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:13.468013    2555 generic.go:356] "Generic (PLEG): container finished" podID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerID="2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608" exitCode=0
Sep 05 02:21:13.468023 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:13.468019    2555 generic.go:356] "Generic (PLEG): container finished" podID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerID="84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01" exitCode=0
Sep 05 02:21:13.468023 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:13.468025    2555 generic.go:356] "Generic (PLEG): container finished" podID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerID="c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1" exitCode=0
Sep 05 02:21:13.468023 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:13.468031    2555 generic.go:356] "Generic (PLEG): container finished" podID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerID="ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40" exitCode=0
Sep 05 02:21:13.468317 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:13.468058    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerDied","Data":"8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148"}
Sep 05 02:21:13.468317 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:13.468129    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerDied","Data":"2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608"}
Sep 05 02:21:13.468317 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:13.468158    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerDied","Data":"84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01"}
Sep 05 02:21:13.468317 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:13.468183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerDied","Data":"c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1"}
Sep 05 02:21:13.468317 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:13.468205    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerDied","Data":"ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40"}
Sep 05 02:21:13.989207 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:13.989180    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.046343 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046265    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-cluster-tls-config\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046343 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046310    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-metrics-client-ca\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046565 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046353    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-out\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046565 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046380    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046565 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046404    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-web-config\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046565 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046429    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-main-db\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046565 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046456    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-trusted-ca-bundle\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046565 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046486    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-web\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046565 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046524    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-tls-assets\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046898 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046586    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-main-tls\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046898 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046620    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j72d\" (UniqueName: \"kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-kube-api-access-6j72d\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046898 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046646    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-volume\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046898 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046689    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\" (UID: \"81b5fc89-3dfe-4b5d-9787-cceb72925fb7\") "
Sep 05 02:21:14.046898 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046775    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:21:14.046898 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046826    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:14.047187 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046964    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-metrics-client-ca\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.047187 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.046986    2555 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-main-db\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.047555 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.047404    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:14.049216 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.049182    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:14.049638 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.049508    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:14.049887 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.049854    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:14.050170 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.050133    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:14.050261 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.050228    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-out" (OuterVolumeSpecName: "config-out") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:21:14.050449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.050427    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-kube-api-access-6j72d" (OuterVolumeSpecName: "kube-api-access-6j72d") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "kube-api-access-6j72d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:14.051355 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.051332    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:14.051980 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.051953    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-volume" (OuterVolumeSpecName: "config-volume") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:14.054586 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.054450    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:14.062093 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.062027    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-web-config" (OuterVolumeSpecName: "web-config") pod "81b5fc89-3dfe-4b5d-9787-cceb72925fb7" (UID: "81b5fc89-3dfe-4b5d-9787-cceb72925fb7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:14.147599 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.147559    2555 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.147599 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.147589    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.147599 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.147601    2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-tls-assets\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.147599 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.147610    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-main-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.147861 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.147621    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j72d\" (UniqueName: \"kubernetes.io/projected/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-kube-api-access-6j72d\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.147861 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.147629    2555 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-volume\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.147861 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.147638    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.147861 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.147646    2555 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-cluster-tls-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.147861 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.147654    2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-config-out\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.147861 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.147662    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.147861 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.147671    2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81b5fc89-3dfe-4b5d-9787-cceb72925fb7-web-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:21:14.474705 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.474672    2555 generic.go:356] "Generic (PLEG): container finished" podID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerID="405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05" exitCode=0
Sep 05 02:21:14.474869 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.474738    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerDied","Data":"405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05"}
Sep 05 02:21:14.474869 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.474765    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.474869 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.474841    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81b5fc89-3dfe-4b5d-9787-cceb72925fb7","Type":"ContainerDied","Data":"8fe06ae0ab8fe70b2d15743ae794f82b6bcbb88ca37f5b325f32d5e10691fcf4"}
Sep 05 02:21:14.475038 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.474887    2555 scope.go:117] "RemoveContainer" containerID="8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148"
Sep 05 02:21:14.482332 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.482232    2555 scope.go:117] "RemoveContainer" containerID="2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608"
Sep 05 02:21:14.492033 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.492015    2555 scope.go:117] "RemoveContainer" containerID="84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01"
Sep 05 02:21:14.507665 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.507622    2555 scope.go:117] "RemoveContainer" containerID="405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05"
Sep 05 02:21:14.509902 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.509877    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:21:14.510960 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.510935    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:21:14.515502 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.515478    2555 scope.go:117] "RemoveContainer" containerID="c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1"
Sep 05 02:21:14.522587 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.522568    2555 scope.go:117] "RemoveContainer" containerID="ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40"
Sep 05 02:21:14.527965 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.527945    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:21:14.528192 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528180    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy"
Sep 05 02:21:14.528248 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528193    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy"
Sep 05 02:21:14.528248 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528204    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy-web"
Sep 05 02:21:14.528248 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528210    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy-web"
Sep 05 02:21:14.528248 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528218    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="alertmanager"
Sep 05 02:21:14.528248 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528224    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="alertmanager"
Sep 05 02:21:14.528248 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528231    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1edddb2-de81-436c-bd1b-720e994f6e70" containerName="registry"
Sep 05 02:21:14.528248 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528236    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1edddb2-de81-436c-bd1b-720e994f6e70" containerName="registry"
Sep 05 02:21:14.528248 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528245    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="init-config-reloader"
Sep 05 02:21:14.528248 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528251    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="init-config-reloader"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528259    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="config-reloader"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528264    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="config-reloader"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528275    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="prom-label-proxy"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528281    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="prom-label-proxy"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528289    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy-metric"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528295    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy-metric"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528332    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="config-reloader"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528339    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy-metric"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528345    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy-web"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528350    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="kube-rbac-proxy"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528354    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="prom-label-proxy"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528361    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" containerName="alertmanager"
Sep 05 02:21:14.528576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.528365    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1edddb2-de81-436c-bd1b-720e994f6e70" containerName="registry"
Sep 05 02:21:14.529694 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.529674    2555 scope.go:117] "RemoveContainer" containerID="11fdf70555c2021766171ca4a0e148bb7720405502cb5aeaa86780e866d295d3"
Sep 05 02:21:14.533288 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.532914    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.535412 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.535392    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 05 02:21:14.535501 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.535480    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 05 02:21:14.535645 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.535480    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mg4dt\""
Sep 05 02:21:14.535694 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.535678    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 05 02:21:14.535765 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.535732    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 05 02:21:14.536323 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.536273    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 05 02:21:14.536525 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.536330    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 05 02:21:14.536525 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.536367    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 05 02:21:14.536525 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.536334    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 05 02:21:14.538386 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.538372    2555 scope.go:117] "RemoveContainer" containerID="8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148"
Sep 05 02:21:14.538732 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:21:14.538713    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148\": container with ID starting with 8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148 not found: ID does not exist" containerID="8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148"
Sep 05 02:21:14.538846 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.538741    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148"} err="failed to get container status \"8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148\": rpc error: code = NotFound desc = could not find container \"8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148\": container with ID starting with 8b198f6fc7293edfc71690dd0fb9ce1b8ebda5e92f214fa4d20a338a7749d148 not found: ID does not exist"
Sep 05 02:21:14.538846 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.538766    2555 scope.go:117] "RemoveContainer" containerID="2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608"
Sep 05 02:21:14.539046 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:21:14.539027    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608\": container with ID starting with 2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608 not found: ID does not exist" containerID="2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608"
Sep 05 02:21:14.539085 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.539053    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608"} err="failed to get container status \"2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608\": rpc error: code = NotFound desc = could not find container \"2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608\": container with ID starting with 2790cbed8101a832717dd79b9d83aea070308072179c196f5af1961e5c693608 not found: ID does not exist"
Sep 05 02:21:14.539085 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.539072    2555 scope.go:117] "RemoveContainer" containerID="84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01"
Sep 05 02:21:14.539385 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:21:14.539369    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01\": container with ID starting with 84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01 not found: ID does not exist" containerID="84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01"
Sep 05 02:21:14.539466 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.539391    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01"} err="failed to get container status \"84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01\": rpc error: code = NotFound desc = could not find container \"84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01\": container with ID starting with 84a4d882d0f2040042736803d42a700d8e43e188dd4921ae7f4f5407f5fd7c01 not found: ID does not exist"
Sep 05 02:21:14.539466 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.539408    2555 scope.go:117] "RemoveContainer" containerID="405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05"
Sep 05 02:21:14.539836 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:21:14.539806    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05\": container with ID starting with 405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05 not found: ID does not exist" containerID="405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05"
Sep 05 02:21:14.539980 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.539842    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05"} err="failed to get container status \"405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05\": rpc error: code = NotFound desc = could not find container \"405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05\": container with ID starting with 405acbe211371f978c775bd3f7716f080526885bb4d8dd49794e12d72e627b05 not found: ID does not exist"
Sep 05 02:21:14.539980 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.539863    2555 scope.go:117] "RemoveContainer" containerID="c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1"
Sep 05 02:21:14.540146 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:21:14.540128    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1\": container with ID starting with c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1 not found: ID does not exist" containerID="c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1"
Sep 05 02:21:14.540239 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.540149    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1"} err="failed to get container status \"c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1\": rpc error: code = NotFound desc = could not find container \"c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1\": container with ID starting with c9e9b64e91b1faf7c41cf9197a1d248e09b9a0f26e45edfe1d224383f42c60c1 not found: ID does not exist"
Sep 05 02:21:14.540239 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.540165    2555 scope.go:117] "RemoveContainer" containerID="ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40"
Sep 05 02:21:14.540468 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:21:14.540442    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40\": container with ID starting with ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40 not found: ID does not exist" containerID="ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40"
Sep 05 02:21:14.540529 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.540477    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40"} err="failed to get container status \"ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40\": rpc error: code = NotFound desc = could not find container \"ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40\": container with ID starting with ad3ea4d6a3cec1d3e7d3b9558c5a213c9edd25693a882dd5b32567d7bbf98a40 not found: ID does not exist"
Sep 05 02:21:14.540529 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.540500    2555 scope.go:117] "RemoveContainer" containerID="11fdf70555c2021766171ca4a0e148bb7720405502cb5aeaa86780e866d295d3"
Sep 05 02:21:14.541107 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:21:14.540931    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fdf70555c2021766171ca4a0e148bb7720405502cb5aeaa86780e866d295d3\": container with ID starting with 11fdf70555c2021766171ca4a0e148bb7720405502cb5aeaa86780e866d295d3 not found: ID does not exist" containerID="11fdf70555c2021766171ca4a0e148bb7720405502cb5aeaa86780e866d295d3"
Sep 05 02:21:14.541107 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.540965    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fdf70555c2021766171ca4a0e148bb7720405502cb5aeaa86780e866d295d3"} err="failed to get container status \"11fdf70555c2021766171ca4a0e148bb7720405502cb5aeaa86780e866d295d3\": rpc error: code = NotFound desc = could not find container \"11fdf70555c2021766171ca4a0e148bb7720405502cb5aeaa86780e866d295d3\": container with ID starting with 11fdf70555c2021766171ca4a0e148bb7720405502cb5aeaa86780e866d295d3 not found: ID does not exist"
Sep 05 02:21:14.541481 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.541459    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 05 02:21:14.549993 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.549974    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:21:14.550222 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550203    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550263 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfcqz\" (UniqueName: \"kubernetes.io/projected/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-kube-api-access-hfcqz\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550308 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550268    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550308 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550305    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550370 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550321    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-web-config\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550370 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550351    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550430 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550383    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550465 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550431    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550465 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550450    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-config-volume\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550522 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550469    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550522 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550490    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550605 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550528    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.550605 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.550585    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-config-out\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651064 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651029    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651064 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651070    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651288 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651094    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-config-out\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651288 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651227    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651288 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651277    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfcqz\" (UniqueName: \"kubernetes.io/projected/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-kube-api-access-hfcqz\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651437 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651319    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651437 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651356    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651437 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-web-config\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651625 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651438    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651625 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651457    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651625 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651625 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651562    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-config-volume\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651625 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651604    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.651939 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.651915    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.652372 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.652305    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.653070 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.653037    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.653954 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.653928    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-config-out\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.654255 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.654225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.654438 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.654414    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.654516 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.654502    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.654749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.654727    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-web-config\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.654938 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.654918    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.655477 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.655453    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.655679 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.655662    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.655796 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.655779    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-config-volume\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.658755 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.658732    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfcqz\" (UniqueName: \"kubernetes.io/projected/bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6-kube-api-access-hfcqz\") pod \"alertmanager-main-0\" (UID: \"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:14.844042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:14.843995    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:21:15.124084 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:15.124052    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:21:15.127381 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:21:15.127352    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb9e01c_a94b_4ce9_9690_5e16a6fd5ce6.slice/crio-a208405bfa162c6375fc138e8cbd2ea58b7d4827013d6f514aa7fd3e9c7d89cf WatchSource:0}: Error finding container a208405bfa162c6375fc138e8cbd2ea58b7d4827013d6f514aa7fd3e9c7d89cf: Status 404 returned error can't find the container with id a208405bfa162c6375fc138e8cbd2ea58b7d4827013d6f514aa7fd3e9c7d89cf
Sep 05 02:21:15.480751 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:15.480674    2555 generic.go:356] "Generic (PLEG): container finished" podID="bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6" containerID="cf641aca212044d54bd75d19413e8ed1a73e325bf5428fbba10eadeb754260e7" exitCode=0
Sep 05 02:21:15.480909 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:15.480764    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6","Type":"ContainerDied","Data":"cf641aca212044d54bd75d19413e8ed1a73e325bf5428fbba10eadeb754260e7"}
Sep 05 02:21:15.480909 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:15.480831    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6","Type":"ContainerStarted","Data":"a208405bfa162c6375fc138e8cbd2ea58b7d4827013d6f514aa7fd3e9c7d89cf"}
Sep 05 02:21:15.543923 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:15.543900    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b5fc89-3dfe-4b5d-9787-cceb72925fb7" path="/var/lib/kubelet/pods/81b5fc89-3dfe-4b5d-9787-cceb72925fb7/volumes"
Sep 05 02:21:16.488443 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.488399    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6","Type":"ContainerStarted","Data":"d5da8162fbf4fd97ed39e9ac122417e8ccfde549dcdfee263f307c9dcb7d1de4"}
Sep 05 02:21:16.488881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.488497    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6","Type":"ContainerStarted","Data":"f7e30fd84e9766bd49af9a7a433f9846e6027a2734c7633c3f7bc3f9c55fbdb2"}
Sep 05 02:21:16.488881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.488526    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6","Type":"ContainerStarted","Data":"6be90ab165437ab25551c6f36af6bca797a606035c54b6181daa0e5d34fdea43"}
Sep 05 02:21:16.488881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.488570    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6","Type":"ContainerStarted","Data":"9f59c1da60d84ce225609cda569fcbdeccdc52666f80c1285e1394810d62b690"}
Sep 05 02:21:16.488881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.488594    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6","Type":"ContainerStarted","Data":"e2a1e2dfbaabe5742d854be3e3dc6677b04bfe6164035342d11e428092fe8a6c"}
Sep 05 02:21:16.488881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.488626    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6","Type":"ContainerStarted","Data":"f082a16f5d43f3b9e6e1f1f504900cc6e5831607a8a94c05aecb7d0ce48fff7a"}
Sep 05 02:21:16.518433 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.518382    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.502982958 podStartE2EDuration="2.518365967s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.383009ms" isStatefulPod=false podCreationTimestamp="2026-09-05 02:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:16.51797963 +0000 UTC m=+197.526475821" watchObservedRunningTime="2026-09-05 02:21:16.518365967 +0000 UTC m=+197.526862107"
Sep 05 02:21:16.826607 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.826506    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"]
Sep 05 02:21:16.829094 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.829070    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.832101 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.832076    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 05 02:21:16.832268 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.832249    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-85jwz\""
Sep 05 02:21:16.832337 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.832308    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 05 02:21:16.832337 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.832186    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 05 02:21:16.832449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.832111    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 05 02:21:16.832449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.832135    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 05 02:21:16.838328 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.838299    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 05 02:21:16.846412 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.846384    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"]
Sep 05 02:21:16.869296 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.869267    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.869443 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.869303    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc60118c-39a7-4b03-9a78-8f1556aa58ca-metrics-client-ca\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.869443 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.869322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct7lt\" (UniqueName: \"kubernetes.io/projected/cc60118c-39a7-4b03-9a78-8f1556aa58ca-kube-api-access-ct7lt\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.869443 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.869352    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-federate-client-tls\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.869443 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.869419    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-secret-telemeter-client\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.869611 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.869449    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc60118c-39a7-4b03-9a78-8f1556aa58ca-telemeter-trusted-ca-bundle\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.869611 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.869469    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc60118c-39a7-4b03-9a78-8f1556aa58ca-serving-certs-ca-bundle\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.869611 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.869524    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-telemeter-client-tls\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.970452 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.970414    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-telemeter-client-tls\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.970609 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.970471    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.970609 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.970501    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc60118c-39a7-4b03-9a78-8f1556aa58ca-metrics-client-ca\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.970609 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.970519    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct7lt\" (UniqueName: \"kubernetes.io/projected/cc60118c-39a7-4b03-9a78-8f1556aa58ca-kube-api-access-ct7lt\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.970609 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.970560    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-federate-client-tls\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.970609 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.970587    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-secret-telemeter-client\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.970872 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.970743    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc60118c-39a7-4b03-9a78-8f1556aa58ca-telemeter-trusted-ca-bundle\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.970872 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.970788    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc60118c-39a7-4b03-9a78-8f1556aa58ca-serving-certs-ca-bundle\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.971398 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.971370    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc60118c-39a7-4b03-9a78-8f1556aa58ca-metrics-client-ca\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.971509 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.971402    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc60118c-39a7-4b03-9a78-8f1556aa58ca-serving-certs-ca-bundle\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.971616 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.971591    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc60118c-39a7-4b03-9a78-8f1556aa58ca-telemeter-trusted-ca-bundle\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.973145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.973123    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-secret-telemeter-client\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.973295 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.973272    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-telemeter-client-tls\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.973753 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.973731    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-federate-client-tls\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.973964 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.973945    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc60118c-39a7-4b03-9a78-8f1556aa58ca-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:16.978773 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:16.978750    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct7lt\" (UniqueName: \"kubernetes.io/projected/cc60118c-39a7-4b03-9a78-8f1556aa58ca-kube-api-access-ct7lt\") pod \"telemeter-client-789c6dcd94-nqlnn\" (UID: \"cc60118c-39a7-4b03-9a78-8f1556aa58ca\") " pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:17.141204 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:17.141158    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"
Sep 05 02:21:17.431525 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:17.431454    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-789c6dcd94-nqlnn"]
Sep 05 02:21:17.494967 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:17.494932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn" event={"ID":"cc60118c-39a7-4b03-9a78-8f1556aa58ca","Type":"ContainerStarted","Data":"0742534bc3dcfdea21979ac05821315333b012c5daf348cc1715b6594a16d944"}
Sep 05 02:21:19.506696 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:19.506598    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn" event={"ID":"cc60118c-39a7-4b03-9a78-8f1556aa58ca","Type":"ContainerStarted","Data":"539bae5dffd6305f9c59041c864bea56ddf5474b02406e15f174d7d6983eb4d5"}
Sep 05 02:21:19.507098 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:19.506701    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn" event={"ID":"cc60118c-39a7-4b03-9a78-8f1556aa58ca","Type":"ContainerStarted","Data":"5ef7c0463c22f77c32c6688d6a612866cb164405b92be7fb62fc1f4aaf4d62f4"}
Sep 05 02:21:19.507098 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:19.506737    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn" event={"ID":"cc60118c-39a7-4b03-9a78-8f1556aa58ca","Type":"ContainerStarted","Data":"477e4af3147b20545c21a2732867e83912a791c4b1940626de464995b2f27604"}
Sep 05 02:21:19.529750 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:21:19.529691    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-789c6dcd94-nqlnn" podStartSLOduration=1.768972707 podStartE2EDuration="3.52967118s" totalImagesPullingTime="1.760698473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:19.528950771 +0000 UTC m=+200.537446922" watchObservedRunningTime="2026-09-05 02:21:19.52967118 +0000 UTC m=+200.538167315"
Sep 05 02:22:59.414377 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:22:59.414338    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:22:59.418836 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:22:59.418807    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:22:59.425402 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:22:59.425381    2555 kubelet.go:1648] "Image garbage collection succeeded"
Sep 05 02:24:07.524828 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.524747    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"]
Sep 05 02:24:07.528217 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.528195    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:07.530958 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.530935    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 02:24:07.531081 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.530933    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-78n45\""
Sep 05 02:24:07.531693 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.531670    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 02:24:07.537055 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.537031    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"]
Sep 05 02:24:07.686479 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.686434    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:07.686726 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.686575    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-488pc\" (UniqueName: \"kubernetes.io/projected/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-kube-api-access-488pc\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:07.686726 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.686640    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:07.787940 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.787850    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-488pc\" (UniqueName: \"kubernetes.io/projected/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-kube-api-access-488pc\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:07.787940 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.787899    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:07.788131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.788017    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:07.788319 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.788296    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:07.788425 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.788336    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:07.795480 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.795460    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-488pc\" (UniqueName: \"kubernetes.io/projected/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-kube-api-access-488pc\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:07.838483 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:07.838452    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:08.107734 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:08.107700    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"]
Sep 05 02:24:08.109830 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:24:08.109805    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae790f2_d3e7_4a4c_b4c1_c42b5b8bddbe.slice/crio-6e029049e5d5b42f5e2826315b15f070098643f58806cfda3fc8b42bb857afe4 WatchSource:0}: Error finding container 6e029049e5d5b42f5e2826315b15f070098643f58806cfda3fc8b42bb857afe4: Status 404 returned error can't find the container with id 6e029049e5d5b42f5e2826315b15f070098643f58806cfda3fc8b42bb857afe4
Sep 05 02:24:08.111687 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:08.111660    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:24:08.140581 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:08.140544    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2" event={"ID":"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe","Type":"ContainerStarted","Data":"6e029049e5d5b42f5e2826315b15f070098643f58806cfda3fc8b42bb857afe4"}
Sep 05 02:24:14.169051 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:14.169011    2555 generic.go:356] "Generic (PLEG): container finished" podID="1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" containerID="c07089eec6fd03b744551cc75b35500f2580d1895876b1ba1da019a9397ea825" exitCode=0
Sep 05 02:24:14.169565 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:14.169058    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2" event={"ID":"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe","Type":"ContainerDied","Data":"c07089eec6fd03b744551cc75b35500f2580d1895876b1ba1da019a9397ea825"}
Sep 05 02:24:16.178444 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:16.178362    2555 generic.go:356] "Generic (PLEG): container finished" podID="1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" containerID="39ecd08f5f1d2afabf7e2450ffb8d508605c8f7bb77642bec61ee6e2494020b1" exitCode=0
Sep 05 02:24:16.178830 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:16.178444    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2" event={"ID":"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe","Type":"ContainerDied","Data":"39ecd08f5f1d2afabf7e2450ffb8d508605c8f7bb77642bec61ee6e2494020b1"}
Sep 05 02:24:22.207654 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:22.207617    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2" event={"ID":"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe","Type":"ContainerStarted","Data":"0b8582cd1b720cfa1c2dde05798677ea23cd32fac0f2072179a7c39200b7e9cd"}
Sep 05 02:24:22.226431 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:22.226380    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2" podStartSLOduration=2.454710053 podStartE2EDuration="15.226365199s" totalImagesPullingTime="12.771655146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:22.225177332 +0000 UTC m=+383.233673497" watchObservedRunningTime="2026-09-05 02:24:22.226365199 +0000 UTC m=+383.234861337"
Sep 05 02:24:23.213584 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:23.213551    2555 generic.go:356] "Generic (PLEG): container finished" podID="1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" containerID="0b8582cd1b720cfa1c2dde05798677ea23cd32fac0f2072179a7c39200b7e9cd" exitCode=0
Sep 05 02:24:23.213957 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:23.213636    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2" event={"ID":"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe","Type":"ContainerDied","Data":"0b8582cd1b720cfa1c2dde05798677ea23cd32fac0f2072179a7c39200b7e9cd"}
Sep 05 02:24:24.344892 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:24.344868    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:24.529308 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:24.529217    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-util\") pod \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") "
Sep 05 02:24:24.529308 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:24.529275    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-bundle\") pod \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") "
Sep 05 02:24:24.529493 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:24.529330    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-488pc\" (UniqueName: \"kubernetes.io/projected/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-kube-api-access-488pc\") pod \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\" (UID: \"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe\") "
Sep 05 02:24:24.529944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:24.529918    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-bundle" (OuterVolumeSpecName: "bundle") pod "1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" (UID: "1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:24:24.531463 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:24.531436    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-kube-api-access-488pc" (OuterVolumeSpecName: "kube-api-access-488pc") pod "1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" (UID: "1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe"). InnerVolumeSpecName "kube-api-access-488pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:24:24.533348 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:24.533321    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-util" (OuterVolumeSpecName: "util") pod "1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" (UID: "1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:24:24.630336 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:24.630295    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-bundle\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:24:24.630336 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:24.630329    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-488pc\" (UniqueName: \"kubernetes.io/projected/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-kube-api-access-488pc\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:24:24.630336 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:24.630339    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe-util\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:24:25.223224 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:25.223186    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2" event={"ID":"1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe","Type":"ContainerDied","Data":"6e029049e5d5b42f5e2826315b15f070098643f58806cfda3fc8b42bb857afe4"}
Sep 05 02:24:25.223392 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:25.223241    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asfhz2"
Sep 05 02:24:25.223392 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:25.223243    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e029049e5d5b42f5e2826315b15f070098643f58806cfda3fc8b42bb857afe4"
Sep 05 02:24:31.075488 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.075453    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"]
Sep 05 02:24:31.075888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.075785    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" containerName="extract"
Sep 05 02:24:31.075888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.075796    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" containerName="extract"
Sep 05 02:24:31.075888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.075805    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" containerName="pull"
Sep 05 02:24:31.075888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.075810    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" containerName="pull"
Sep 05 02:24:31.075888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.075824    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" containerName="util"
Sep 05 02:24:31.075888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.075840    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" containerName="util"
Sep 05 02:24:31.075888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.075885    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae790f2-d3e7-4a4c-b4c1-c42b5b8bddbe" containerName="extract"
Sep 05 02:24:31.111832 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.111798    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"]
Sep 05 02:24:31.111992 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.111892    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"
Sep 05 02:24:31.116064 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.116042    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 05 02:24:31.116064 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.116052    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 05 02:24:31.116229 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.116207    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 05 02:24:31.116447 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.116422    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-x94bs\""
Sep 05 02:24:31.173642 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.173608    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx52x\" (UniqueName: \"kubernetes.io/projected/74661d8c-e2de-4a76-8d6a-0f855a14853a-kube-api-access-hx52x\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-h2qmx\" (UID: \"74661d8c-e2de-4a76-8d6a-0f855a14853a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"
Sep 05 02:24:31.173814 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.173687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/74661d8c-e2de-4a76-8d6a-0f855a14853a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-h2qmx\" (UID: \"74661d8c-e2de-4a76-8d6a-0f855a14853a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"
Sep 05 02:24:31.274875 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.274838    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx52x\" (UniqueName: \"kubernetes.io/projected/74661d8c-e2de-4a76-8d6a-0f855a14853a-kube-api-access-hx52x\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-h2qmx\" (UID: \"74661d8c-e2de-4a76-8d6a-0f855a14853a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"
Sep 05 02:24:31.275019 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.274927    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/74661d8c-e2de-4a76-8d6a-0f855a14853a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-h2qmx\" (UID: \"74661d8c-e2de-4a76-8d6a-0f855a14853a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"
Sep 05 02:24:31.277236 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.277217    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/74661d8c-e2de-4a76-8d6a-0f855a14853a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-h2qmx\" (UID: \"74661d8c-e2de-4a76-8d6a-0f855a14853a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"
Sep 05 02:24:31.284489 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.284463    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx52x\" (UniqueName: \"kubernetes.io/projected/74661d8c-e2de-4a76-8d6a-0f855a14853a-kube-api-access-hx52x\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-h2qmx\" (UID: \"74661d8c-e2de-4a76-8d6a-0f855a14853a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"
Sep 05 02:24:31.421640 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.421603    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"
Sep 05 02:24:31.726969 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:31.726937    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"]
Sep 05 02:24:31.728249 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:24:31.728206    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74661d8c_e2de_4a76_8d6a_0f855a14853a.slice/crio-54d27c1caf64fe32b5923a69d7de8dc9a367d9bad3c411a262805785d6355f6f WatchSource:0}: Error finding container 54d27c1caf64fe32b5923a69d7de8dc9a367d9bad3c411a262805785d6355f6f: Status 404 returned error can't find the container with id 54d27c1caf64fe32b5923a69d7de8dc9a367d9bad3c411a262805785d6355f6f
Sep 05 02:24:32.254235 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:32.254196    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx" event={"ID":"74661d8c-e2de-4a76-8d6a-0f855a14853a","Type":"ContainerStarted","Data":"54d27c1caf64fe32b5923a69d7de8dc9a367d9bad3c411a262805785d6355f6f"}
Sep 05 02:24:35.275830 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:35.275792    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx" event={"ID":"74661d8c-e2de-4a76-8d6a-0f855a14853a","Type":"ContainerStarted","Data":"1f967b086382a01f15a5ff065ffe0f5dca13b75c0518774a6862f8c3f54eee5e"}
Sep 05 02:24:35.276339 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:35.275970    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"
Sep 05 02:24:35.300207 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:35.300121    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx" podStartSLOduration=0.993796396 podStartE2EDuration="4.300105585s" totalImagesPullingTime="3.306309189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:35.298096348 +0000 UTC m=+396.306592486" watchObservedRunningTime="2026-09-05 02:24:35.300105585 +0000 UTC m=+396.308601724"
Sep 05 02:24:36.011472 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.011439    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-c4xfv"]
Sep 05 02:24:36.026697 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.026668    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:36.029340 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.029313    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-s9pjp\""
Sep 05 02:24:36.030033 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.030012    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 05 02:24:36.030823 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.030804    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 05 02:24:36.037775 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.037750    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-c4xfv"]
Sep 05 02:24:36.112636 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.112600    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhcs4\" (UniqueName: \"kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-kube-api-access-nhcs4\") pod \"keda-operator-579c8958f7-c4xfv\" (UID: \"1430b95c-646e-4cf9-a5e2-773fa5f87ba9\") " pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:36.112636 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.112636    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-certificates\") pod \"keda-operator-579c8958f7-c4xfv\" (UID: \"1430b95c-646e-4cf9-a5e2-773fa5f87ba9\") " pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:36.112851 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.112653    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-cabundle0\") pod \"keda-operator-579c8958f7-c4xfv\" (UID: \"1430b95c-646e-4cf9-a5e2-773fa5f87ba9\") " pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:36.213915 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.213875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-certificates\") pod \"keda-operator-579c8958f7-c4xfv\" (UID: \"1430b95c-646e-4cf9-a5e2-773fa5f87ba9\") " pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:36.214093 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.213921    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-cabundle0\") pod \"keda-operator-579c8958f7-c4xfv\" (UID: \"1430b95c-646e-4cf9-a5e2-773fa5f87ba9\") " pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:36.214093 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.214008    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhcs4\" (UniqueName: \"kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-kube-api-access-nhcs4\") pod \"keda-operator-579c8958f7-c4xfv\" (UID: \"1430b95c-646e-4cf9-a5e2-773fa5f87ba9\") " pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:36.214093 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.214018    2555 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 05 02:24:36.214093 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.214039    2555 secret.go:281] references non-existent secret key: ca.crt
Sep 05 02:24:36.214093 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.214047    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 05 02:24:36.214093 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.214061    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-c4xfv: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 05 02:24:36.214354 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.214123    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-certificates podName:1430b95c-646e-4cf9-a5e2-773fa5f87ba9 nodeName:}" failed. No retries permitted until 2026-09-05 02:24:36.714106573 +0000 UTC m=+397.722602690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-certificates") pod "keda-operator-579c8958f7-c4xfv" (UID: "1430b95c-646e-4cf9-a5e2-773fa5f87ba9") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 05 02:24:36.214607 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.214589    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-cabundle0\") pod \"keda-operator-579c8958f7-c4xfv\" (UID: \"1430b95c-646e-4cf9-a5e2-773fa5f87ba9\") " pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:36.226553 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.226514    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhcs4\" (UniqueName: \"kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-kube-api-access-nhcs4\") pod \"keda-operator-579c8958f7-c4xfv\" (UID: \"1430b95c-646e-4cf9-a5e2-773fa5f87ba9\") " pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:36.305332 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.305263    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"]
Sep 05 02:24:36.325873 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.325851    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:36.328509 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.328479    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"]
Sep 05 02:24:36.328740 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.328720    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 05 02:24:36.416404 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.416366    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/34b0b901-dfa6-406f-be5f-f3efae30836a-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gq79w\" (UID: \"34b0b901-dfa6-406f-be5f-f3efae30836a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:36.416604 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.416424    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/34b0b901-dfa6-406f-be5f-f3efae30836a-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gq79w\" (UID: \"34b0b901-dfa6-406f-be5f-f3efae30836a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:36.416604 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.416468    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjqpp\" (UniqueName: \"kubernetes.io/projected/34b0b901-dfa6-406f-be5f-f3efae30836a-kube-api-access-hjqpp\") pod \"keda-metrics-apiserver-77857b7b98-gq79w\" (UID: \"34b0b901-dfa6-406f-be5f-f3efae30836a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:36.517504 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.517471    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/34b0b901-dfa6-406f-be5f-f3efae30836a-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gq79w\" (UID: \"34b0b901-dfa6-406f-be5f-f3efae30836a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:36.517678 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.517516    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/34b0b901-dfa6-406f-be5f-f3efae30836a-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gq79w\" (UID: \"34b0b901-dfa6-406f-be5f-f3efae30836a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:36.517678 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.517561    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjqpp\" (UniqueName: \"kubernetes.io/projected/34b0b901-dfa6-406f-be5f-f3efae30836a-kube-api-access-hjqpp\") pod \"keda-metrics-apiserver-77857b7b98-gq79w\" (UID: \"34b0b901-dfa6-406f-be5f-f3efae30836a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:36.517678 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.517620    2555 secret.go:281] references non-existent secret key: tls.crt
Sep 05 02:24:36.517678 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.517642    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 05 02:24:36.517678 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.517659    2555 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 05 02:24:36.517678 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.517673    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 05 02:24:36.517861 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.517743    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34b0b901-dfa6-406f-be5f-f3efae30836a-certificates podName:34b0b901-dfa6-406f-be5f-f3efae30836a nodeName:}" failed. No retries permitted until 2026-09-05 02:24:37.017730342 +0000 UTC m=+398.026226459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/34b0b901-dfa6-406f-be5f-f3efae30836a-certificates") pod "keda-metrics-apiserver-77857b7b98-gq79w" (UID: "34b0b901-dfa6-406f-be5f-f3efae30836a") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 05 02:24:36.517928 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.517911    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/34b0b901-dfa6-406f-be5f-f3efae30836a-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gq79w\" (UID: \"34b0b901-dfa6-406f-be5f-f3efae30836a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:36.541267 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.541234    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjqpp\" (UniqueName: \"kubernetes.io/projected/34b0b901-dfa6-406f-be5f-f3efae30836a-kube-api-access-hjqpp\") pod \"keda-metrics-apiserver-77857b7b98-gq79w\" (UID: \"34b0b901-dfa6-406f-be5f-f3efae30836a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:36.719641 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:36.719601    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-certificates\") pod \"keda-operator-579c8958f7-c4xfv\" (UID: \"1430b95c-646e-4cf9-a5e2-773fa5f87ba9\") " pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:36.719811 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.719749    2555 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 05 02:24:36.719811 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.719775    2555 secret.go:281] references non-existent secret key: ca.crt
Sep 05 02:24:36.719811 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.719782    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 05 02:24:36.719811 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.719792    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-c4xfv: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 05 02:24:36.719949 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:24:36.719854    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-certificates podName:1430b95c-646e-4cf9-a5e2-773fa5f87ba9 nodeName:}" failed. No retries permitted until 2026-09-05 02:24:37.719838613 +0000 UTC m=+398.728334729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-certificates") pod "keda-operator-579c8958f7-c4xfv" (UID: "1430b95c-646e-4cf9-a5e2-773fa5f87ba9") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 05 02:24:37.022141 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:37.022058    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/34b0b901-dfa6-406f-be5f-f3efae30836a-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gq79w\" (UID: \"34b0b901-dfa6-406f-be5f-f3efae30836a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:37.024440 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:37.024405    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/34b0b901-dfa6-406f-be5f-f3efae30836a-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gq79w\" (UID: \"34b0b901-dfa6-406f-be5f-f3efae30836a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:37.238640 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:37.238264    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:37.545255 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:37.545226    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"]
Sep 05 02:24:37.545859 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:24:37.545295    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b0b901_dfa6_406f_be5f_f3efae30836a.slice/crio-a180bcea48931f2128c5bc120f31cccf28dbcb7cfd2a2b79e7602ad30f027e1b WatchSource:0}: Error finding container a180bcea48931f2128c5bc120f31cccf28dbcb7cfd2a2b79e7602ad30f027e1b: Status 404 returned error can't find the container with id a180bcea48931f2128c5bc120f31cccf28dbcb7cfd2a2b79e7602ad30f027e1b
Sep 05 02:24:37.730504 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:37.730470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-certificates\") pod \"keda-operator-579c8958f7-c4xfv\" (UID: \"1430b95c-646e-4cf9-a5e2-773fa5f87ba9\") " pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:37.732921 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:37.732894    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1430b95c-646e-4cf9-a5e2-773fa5f87ba9-certificates\") pod \"keda-operator-579c8958f7-c4xfv\" (UID: \"1430b95c-646e-4cf9-a5e2-773fa5f87ba9\") " pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:37.839151 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:37.838991    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:38.142657 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:38.142629    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-c4xfv"]
Sep 05 02:24:38.144453 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:24:38.144419    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1430b95c_646e_4cf9_a5e2_773fa5f87ba9.slice/crio-5a1a4e6cf8d342499a6342ca16c8fefe105b395464e0953a3ecfa24d0c7358b0 WatchSource:0}: Error finding container 5a1a4e6cf8d342499a6342ca16c8fefe105b395464e0953a3ecfa24d0c7358b0: Status 404 returned error can't find the container with id 5a1a4e6cf8d342499a6342ca16c8fefe105b395464e0953a3ecfa24d0c7358b0
Sep 05 02:24:38.294050 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:38.294012    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-c4xfv" event={"ID":"1430b95c-646e-4cf9-a5e2-773fa5f87ba9","Type":"ContainerStarted","Data":"5a1a4e6cf8d342499a6342ca16c8fefe105b395464e0953a3ecfa24d0c7358b0"}
Sep 05 02:24:38.295381 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:38.295354    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w" event={"ID":"34b0b901-dfa6-406f-be5f-f3efae30836a","Type":"ContainerStarted","Data":"a180bcea48931f2128c5bc120f31cccf28dbcb7cfd2a2b79e7602ad30f027e1b"}
Sep 05 02:24:42.317346 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:42.317251    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-c4xfv" event={"ID":"1430b95c-646e-4cf9-a5e2-773fa5f87ba9","Type":"ContainerStarted","Data":"3552a8bd31882b179f2d14057634157605b114ed98a1e625e578850cb6daee4e"}
Sep 05 02:24:42.317775 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:42.317356    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:24:42.318869 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:42.318846    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w" event={"ID":"34b0b901-dfa6-406f-be5f-f3efae30836a","Type":"ContainerStarted","Data":"8303cec071eb5fc16182892b495d4134f2bf77f4eb7137ef87ebf7f4afeb45d2"}
Sep 05 02:24:42.318995 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:42.318890    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:42.335876 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:42.335833    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-c4xfv" podStartSLOduration=3.576464033 podStartE2EDuration="7.335818117s" totalImagesPullingTime="3.759354084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:42.334093949 +0000 UTC m=+403.342590126" watchObservedRunningTime="2026-09-05 02:24:42.335818117 +0000 UTC m=+403.344314256"
Sep 05 02:24:42.352198 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:42.352152    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w" podStartSLOduration=1.994394712 podStartE2EDuration="6.352138639s" totalImagesPullingTime="4.357743927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:42.35120583 +0000 UTC m=+403.359701981" watchObservedRunningTime="2026-09-05 02:24:42.352138639 +0000 UTC m=+403.360634778"
Sep 05 02:24:53.328803 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:53.328771    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gq79w"
Sep 05 02:24:56.283179 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:24:56.283147    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-h2qmx"
Sep 05 02:25:03.326962 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:25:03.326926    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-c4xfv"
Sep 05 02:26:07.569254 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.569222    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"]
Sep 05 02:26:07.572610 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.572591    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"
Sep 05 02:26:07.576983 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.576958    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 05 02:26:07.577662 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.577640    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 05 02:26:07.577766 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.577679    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-phl9b\""
Sep 05 02:26:07.577766 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.577648    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 05 02:26:07.590008 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.589982    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"]
Sep 05 02:26:07.676629 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.676595    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9h2\" (UniqueName: \"kubernetes.io/projected/3809d232-09b8-457f-81a9-d51b5c4d298e-kube-api-access-dd9h2\") pod \"llmisvc-controller-manager-645f9c8fd6-sc9n4\" (UID: \"3809d232-09b8-457f-81a9-d51b5c4d298e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"
Sep 05 02:26:07.676826 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.676704    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3809d232-09b8-457f-81a9-d51b5c4d298e-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-sc9n4\" (UID: \"3809d232-09b8-457f-81a9-d51b5c4d298e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"
Sep 05 02:26:07.778221 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.778186    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9h2\" (UniqueName: \"kubernetes.io/projected/3809d232-09b8-457f-81a9-d51b5c4d298e-kube-api-access-dd9h2\") pod \"llmisvc-controller-manager-645f9c8fd6-sc9n4\" (UID: \"3809d232-09b8-457f-81a9-d51b5c4d298e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"
Sep 05 02:26:07.778407 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.778249    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3809d232-09b8-457f-81a9-d51b5c4d298e-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-sc9n4\" (UID: \"3809d232-09b8-457f-81a9-d51b5c4d298e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"
Sep 05 02:26:07.780672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.780654    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3809d232-09b8-457f-81a9-d51b5c4d298e-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-sc9n4\" (UID: \"3809d232-09b8-457f-81a9-d51b5c4d298e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"
Sep 05 02:26:07.788859 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.788833    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9h2\" (UniqueName: \"kubernetes.io/projected/3809d232-09b8-457f-81a9-d51b5c4d298e-kube-api-access-dd9h2\") pod \"llmisvc-controller-manager-645f9c8fd6-sc9n4\" (UID: \"3809d232-09b8-457f-81a9-d51b5c4d298e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"
Sep 05 02:26:07.882737 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:07.882708    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"
Sep 05 02:26:08.158561 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:08.158519    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"]
Sep 05 02:26:08.160987 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:26:08.160956    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3809d232_09b8_457f_81a9_d51b5c4d298e.slice/crio-cf22e0a98245596bda364a5de5fcff1fb5c49d9e1a8c4403af2e5ed2d7ce2f04 WatchSource:0}: Error finding container cf22e0a98245596bda364a5de5fcff1fb5c49d9e1a8c4403af2e5ed2d7ce2f04: Status 404 returned error can't find the container with id cf22e0a98245596bda364a5de5fcff1fb5c49d9e1a8c4403af2e5ed2d7ce2f04
Sep 05 02:26:08.708327 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:08.708286    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4" event={"ID":"3809d232-09b8-457f-81a9-d51b5c4d298e","Type":"ContainerStarted","Data":"cf22e0a98245596bda364a5de5fcff1fb5c49d9e1a8c4403af2e5ed2d7ce2f04"}
Sep 05 02:26:10.718930 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:10.718892    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4" event={"ID":"3809d232-09b8-457f-81a9-d51b5c4d298e","Type":"ContainerStarted","Data":"eb0004d3fc66a040d9f370058482a0353aea5b93bc1defcf3d4d884b8eaf3966"}
Sep 05 02:26:10.719362 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:10.719007    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"
Sep 05 02:26:10.737810 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:10.737752    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4" podStartSLOduration=1.946937566 podStartE2EDuration="3.737730731s" totalImagesPullingTime="1.790793165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:10.735622682 +0000 UTC m=+491.744118846" watchObservedRunningTime="2026-09-05 02:26:10.737730731 +0000 UTC m=+491.746226876"
Sep 05 02:26:41.725232 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:26:41.725201    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sc9n4"
Sep 05 02:27:59.444626 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:27:59.444600    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:27:59.445607 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:27:59.445588    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:29:07.602958 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.602923    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"]
Sep 05 02:29:07.606267 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.606251    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"
Sep 05 02:29:07.608796 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.608775    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 05 02:29:07.608796 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.608786    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rb75m\""
Sep 05 02:29:07.614402 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.614380    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"]
Sep 05 02:29:07.659134 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.659096    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxjv7\" (UniqueName: \"kubernetes.io/projected/0dbb861a-97d4-48b9-9190-d2534c73f94c-kube-api-access-bxjv7\") pod \"seaweedfs-tls-custom-6644bf5d6c-nhf64\" (UID: \"0dbb861a-97d4-48b9-9190-d2534c73f94c\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"
Sep 05 02:29:07.659134 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.659144    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0dbb861a-97d4-48b9-9190-d2534c73f94c-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-nhf64\" (UID: \"0dbb861a-97d4-48b9-9190-d2534c73f94c\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"
Sep 05 02:29:07.760202 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.760162    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxjv7\" (UniqueName: \"kubernetes.io/projected/0dbb861a-97d4-48b9-9190-d2534c73f94c-kube-api-access-bxjv7\") pod \"seaweedfs-tls-custom-6644bf5d6c-nhf64\" (UID: \"0dbb861a-97d4-48b9-9190-d2534c73f94c\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"
Sep 05 02:29:07.760202 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.760205    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0dbb861a-97d4-48b9-9190-d2534c73f94c-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-nhf64\" (UID: \"0dbb861a-97d4-48b9-9190-d2534c73f94c\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"
Sep 05 02:29:07.760592 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.760576    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0dbb861a-97d4-48b9-9190-d2534c73f94c-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-nhf64\" (UID: \"0dbb861a-97d4-48b9-9190-d2534c73f94c\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"
Sep 05 02:29:07.769754 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.769730    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxjv7\" (UniqueName: \"kubernetes.io/projected/0dbb861a-97d4-48b9-9190-d2534c73f94c-kube-api-access-bxjv7\") pod \"seaweedfs-tls-custom-6644bf5d6c-nhf64\" (UID: \"0dbb861a-97d4-48b9-9190-d2534c73f94c\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"
Sep 05 02:29:07.916349 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:07.916314    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"
Sep 05 02:29:08.190410 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:08.190372    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"]
Sep 05 02:29:08.193255 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:29:08.193227    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbb861a_97d4_48b9_9190_d2534c73f94c.slice/crio-a1318eeedf02b22a6b369af8ba26fa2a5177133898b70f732410f2782a245c97 WatchSource:0}: Error finding container a1318eeedf02b22a6b369af8ba26fa2a5177133898b70f732410f2782a245c97: Status 404 returned error can't find the container with id a1318eeedf02b22a6b369af8ba26fa2a5177133898b70f732410f2782a245c97
Sep 05 02:29:08.194919 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:08.194900    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:29:08.493009 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:08.492927    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64" event={"ID":"0dbb861a-97d4-48b9-9190-d2534c73f94c","Type":"ContainerStarted","Data":"a1318eeedf02b22a6b369af8ba26fa2a5177133898b70f732410f2782a245c97"}
Sep 05 02:29:11.508610 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:11.508573    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64" event={"ID":"0dbb861a-97d4-48b9-9190-d2534c73f94c","Type":"ContainerStarted","Data":"fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702"}
Sep 05 02:29:11.525262 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:11.525206    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64" podStartSLOduration=1.674276025 podStartE2EDuration="4.525189065s" totalImagesPullingTime="2.85091304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:29:11.523958162 +0000 UTC m=+672.532454308" watchObservedRunningTime="2026-09-05 02:29:11.525189065 +0000 UTC m=+672.533685203"
Sep 05 02:29:12.313503 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:12.313468    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"]
Sep 05 02:29:13.517121 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:13.517057    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64" podUID="0dbb861a-97d4-48b9-9190-d2534c73f94c" containerName="seaweedfs-tls-custom" containerID="cri-o://fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702" gracePeriod=30
Sep 05 02:29:14.759201 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:14.759174    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"
Sep 05 02:29:14.924990 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:14.924944    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0dbb861a-97d4-48b9-9190-d2534c73f94c-data\") pod \"0dbb861a-97d4-48b9-9190-d2534c73f94c\" (UID: \"0dbb861a-97d4-48b9-9190-d2534c73f94c\") "
Sep 05 02:29:14.925163 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:14.925014    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxjv7\" (UniqueName: \"kubernetes.io/projected/0dbb861a-97d4-48b9-9190-d2534c73f94c-kube-api-access-bxjv7\") pod \"0dbb861a-97d4-48b9-9190-d2534c73f94c\" (UID: \"0dbb861a-97d4-48b9-9190-d2534c73f94c\") "
Sep 05 02:29:14.926176 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:14.926140    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbb861a-97d4-48b9-9190-d2534c73f94c-data" (OuterVolumeSpecName: "data") pod "0dbb861a-97d4-48b9-9190-d2534c73f94c" (UID: "0dbb861a-97d4-48b9-9190-d2534c73f94c"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:29:14.927177 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:14.927149    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbb861a-97d4-48b9-9190-d2534c73f94c-kube-api-access-bxjv7" (OuterVolumeSpecName: "kube-api-access-bxjv7") pod "0dbb861a-97d4-48b9-9190-d2534c73f94c" (UID: "0dbb861a-97d4-48b9-9190-d2534c73f94c"). InnerVolumeSpecName "kube-api-access-bxjv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:29:15.025766 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:15.025729    2555 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0dbb861a-97d4-48b9-9190-d2534c73f94c-data\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:29:15.025766 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:15.025761    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxjv7\" (UniqueName: \"kubernetes.io/projected/0dbb861a-97d4-48b9-9190-d2534c73f94c-kube-api-access-bxjv7\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:29:15.526381 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:15.526349    2555 generic.go:356] "Generic (PLEG): container finished" podID="0dbb861a-97d4-48b9-9190-d2534c73f94c" containerID="fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702" exitCode=0
Sep 05 02:29:15.526588 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:15.526387    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"
Sep 05 02:29:15.526588 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:15.526425    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64" event={"ID":"0dbb861a-97d4-48b9-9190-d2534c73f94c","Type":"ContainerDied","Data":"fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702"}
Sep 05 02:29:15.526588 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:15.526492    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64" event={"ID":"0dbb861a-97d4-48b9-9190-d2534c73f94c","Type":"ContainerDied","Data":"a1318eeedf02b22a6b369af8ba26fa2a5177133898b70f732410f2782a245c97"}
Sep 05 02:29:15.526588 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:15.526513    2555 scope.go:117] "RemoveContainer" containerID="fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702"
Sep 05 02:29:15.536157 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:15.536139    2555 scope.go:117] "RemoveContainer" containerID="fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702"
Sep 05 02:29:15.536406 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:29:15.536391    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702\": container with ID starting with fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702 not found: ID does not exist" containerID="fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702"
Sep 05 02:29:15.536459 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:15.536414    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702"} err="failed to get container status \"fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702\": rpc error: code = NotFound desc = could not find container \"fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702\": container with ID starting with fbaff2a4ab986c54eb814ccf3fbfedbe1b1df7fb9a27f36ef3d8b8c0fb8b1702 not found: ID does not exist"
Sep 05 02:29:15.547234 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:15.547210    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"]
Sep 05 02:29:15.551331 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:15.551307    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-nhf64"]
Sep 05 02:29:17.543675 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:17.543645    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dbb861a-97d4-48b9-9190-d2534c73f94c" path="/var/lib/kubelet/pods/0dbb861a-97d4-48b9-9190-d2534c73f94c/volumes"
Sep 05 02:29:30.549376 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.549333    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"]
Sep 05 02:29:30.549847 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.549833    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dbb861a-97d4-48b9-9190-d2534c73f94c" containerName="seaweedfs-tls-custom"
Sep 05 02:29:30.549912 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.549852    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dbb861a-97d4-48b9-9190-d2534c73f94c" containerName="seaweedfs-tls-custom"
Sep 05 02:29:30.549963 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.549944    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dbb861a-97d4-48b9-9190-d2534c73f94c" containerName="seaweedfs-tls-custom"
Sep 05 02:29:30.552793 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.552772    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:30.555091 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.555072    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rb75m\""
Sep 05 02:29:30.555397 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.555374    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 05 02:29:30.555487 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.555395    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 05 02:29:30.561962 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.561940    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"]
Sep 05 02:29:30.562203 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.562182    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/85381b15-07d1-433d-bf7b-7e6494eef6d8-data\") pod \"seaweedfs-tls-serving-67b7477c6d-ndq76\" (UID: \"85381b15-07d1-433d-bf7b-7e6494eef6d8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:30.562252 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.562234    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbndm\" (UniqueName: \"kubernetes.io/projected/85381b15-07d1-433d-bf7b-7e6494eef6d8-kube-api-access-rbndm\") pod \"seaweedfs-tls-serving-67b7477c6d-ndq76\" (UID: \"85381b15-07d1-433d-bf7b-7e6494eef6d8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:30.562295 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.562281    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/85381b15-07d1-433d-bf7b-7e6494eef6d8-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-ndq76\" (UID: \"85381b15-07d1-433d-bf7b-7e6494eef6d8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:30.663180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.663145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/85381b15-07d1-433d-bf7b-7e6494eef6d8-data\") pod \"seaweedfs-tls-serving-67b7477c6d-ndq76\" (UID: \"85381b15-07d1-433d-bf7b-7e6494eef6d8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:30.663180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.663187    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbndm\" (UniqueName: \"kubernetes.io/projected/85381b15-07d1-433d-bf7b-7e6494eef6d8-kube-api-access-rbndm\") pod \"seaweedfs-tls-serving-67b7477c6d-ndq76\" (UID: \"85381b15-07d1-433d-bf7b-7e6494eef6d8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:30.663426 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.663215    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/85381b15-07d1-433d-bf7b-7e6494eef6d8-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-ndq76\" (UID: \"85381b15-07d1-433d-bf7b-7e6494eef6d8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:30.663426 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:29:30.663327    2555 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Sep 05 02:29:30.663426 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:29:30.663338    2555 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-ndq76: secret "seaweedfs-tls-serving" not found
Sep 05 02:29:30.663426 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:29:30.663398    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85381b15-07d1-433d-bf7b-7e6494eef6d8-seaweedfs-tls-serving podName:85381b15-07d1-433d-bf7b-7e6494eef6d8 nodeName:}" failed. No retries permitted until 2026-09-05 02:29:31.16338224 +0000 UTC m=+692.171878358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/85381b15-07d1-433d-bf7b-7e6494eef6d8-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-ndq76" (UID: "85381b15-07d1-433d-bf7b-7e6494eef6d8") : secret "seaweedfs-tls-serving" not found
Sep 05 02:29:30.663584 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.663516    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/85381b15-07d1-433d-bf7b-7e6494eef6d8-data\") pod \"seaweedfs-tls-serving-67b7477c6d-ndq76\" (UID: \"85381b15-07d1-433d-bf7b-7e6494eef6d8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:30.671563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:30.671528    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbndm\" (UniqueName: \"kubernetes.io/projected/85381b15-07d1-433d-bf7b-7e6494eef6d8-kube-api-access-rbndm\") pod \"seaweedfs-tls-serving-67b7477c6d-ndq76\" (UID: \"85381b15-07d1-433d-bf7b-7e6494eef6d8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:31.168201 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:31.168163    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/85381b15-07d1-433d-bf7b-7e6494eef6d8-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-ndq76\" (UID: \"85381b15-07d1-433d-bf7b-7e6494eef6d8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:31.170406 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:31.170377    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/85381b15-07d1-433d-bf7b-7e6494eef6d8-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-ndq76\" (UID: \"85381b15-07d1-433d-bf7b-7e6494eef6d8\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:31.462597 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:31.462511    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"
Sep 05 02:29:31.735321 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:31.735300    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-ndq76"]
Sep 05 02:29:31.738005 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:29:31.737980    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85381b15_07d1_433d_bf7b_7e6494eef6d8.slice/crio-9278d0c75129a488146a06106bdaa6ccaeba19e45be3b7c8b670d8ee86402f97 WatchSource:0}: Error finding container 9278d0c75129a488146a06106bdaa6ccaeba19e45be3b7c8b670d8ee86402f97: Status 404 returned error can't find the container with id 9278d0c75129a488146a06106bdaa6ccaeba19e45be3b7c8b670d8ee86402f97
Sep 05 02:29:32.600063 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:32.600023    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76" event={"ID":"85381b15-07d1-433d-bf7b-7e6494eef6d8","Type":"ContainerStarted","Data":"08209b9ef1cba9d0d6cd8fc0462d6398db1aa4d1477f8c05e7565d90bc92ed1c"}
Sep 05 02:29:32.600224 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:32.600081    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76" event={"ID":"85381b15-07d1-433d-bf7b-7e6494eef6d8","Type":"ContainerStarted","Data":"9278d0c75129a488146a06106bdaa6ccaeba19e45be3b7c8b670d8ee86402f97"}
Sep 05 02:29:32.615949 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:29:32.615901    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-ndq76" podStartSLOduration=2.357153416 podStartE2EDuration="2.615886562s" totalImagesPullingTime="258.733146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:29:32.614495372 +0000 UTC m=+693.622991537" watchObservedRunningTime="2026-09-05 02:29:32.615886562 +0000 UTC m=+693.624382700"
Sep 05 02:30:12.528259 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.528218    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"]
Sep 05 02:30:12.532943 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.532925    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.535369 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.535346    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 05 02:30:12.535497 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.535438    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:30:12.535580 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.535500    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 05 02:30:12.535580 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.535521    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:30:12.535580 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.535502    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:30:12.536286 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.536272    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:30:12.536331 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.536273    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:30:12.541553 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.541520    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"]
Sep 05 02:30:12.636609 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.636573    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.636609 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.636610    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.636881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.636630    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll8k5\" (UniqueName: \"kubernetes.io/projected/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-api-access-ll8k5\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.636881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.636712    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.636881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.636795    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.737290 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.737256    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.737501 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.737308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.737501 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.737348    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.737501 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.737366    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.737501 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.737382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8k5\" (UniqueName: \"kubernetes.io/projected/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-api-access-ll8k5\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.737821 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.737800    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.738081 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.738063    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.738185 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.738164    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.739942 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.739911    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.745691 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.745668    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll8k5\" (UniqueName: \"kubernetes.io/projected/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-api-access-ll8k5\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-bxpsx\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:12.844327 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:12.844233    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:13.117001 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:13.116976    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"]
Sep 05 02:30:13.119115 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:30:13.119083    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9f47a6_3545_4856_85fc_7e1060f2ebe1.slice/crio-d990ad09f8a40a489f62dc1c172deff9df5b424a742adf265d0a208921a3341f WatchSource:0}: Error finding container d990ad09f8a40a489f62dc1c172deff9df5b424a742adf265d0a208921a3341f: Status 404 returned error can't find the container with id d990ad09f8a40a489f62dc1c172deff9df5b424a742adf265d0a208921a3341f
Sep 05 02:30:13.786183 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:13.786148    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" event={"ID":"2b9f47a6-3545-4856-85fc-7e1060f2ebe1","Type":"ContainerStarted","Data":"d990ad09f8a40a489f62dc1c172deff9df5b424a742adf265d0a208921a3341f"}
Sep 05 02:30:16.801993 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:16.801957    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" event={"ID":"2b9f47a6-3545-4856-85fc-7e1060f2ebe1","Type":"ContainerStarted","Data":"4776ad298992920d8ca92f7430d11f63d8fd7be0565b93b75fc7b1aae37dd775"}
Sep 05 02:30:20.820032 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:20.819996    2555 generic.go:356] "Generic (PLEG): container finished" podID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerID="4776ad298992920d8ca92f7430d11f63d8fd7be0565b93b75fc7b1aae37dd775" exitCode=0
Sep 05 02:30:20.820415 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:20.820070    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" event={"ID":"2b9f47a6-3545-4856-85fc-7e1060f2ebe1","Type":"ContainerDied","Data":"4776ad298992920d8ca92f7430d11f63d8fd7be0565b93b75fc7b1aae37dd775"}
Sep 05 02:30:42.966392 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:42.966356    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" event={"ID":"2b9f47a6-3545-4856-85fc-7e1060f2ebe1","Type":"ContainerStarted","Data":"f667d5257ae8785bcfdc9e7ff2d2cd548083aff3d9e012db9dc81eedd0d44023"}
Sep 05 02:30:45.981888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:45.981855    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" event={"ID":"2b9f47a6-3545-4856-85fc-7e1060f2ebe1","Type":"ContainerStarted","Data":"2ae20005267c070a51c3505510a7583514c8bd9f3eaf220aca78f289076e2bf2"}
Sep 05 02:30:45.982307 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:45.982082    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:45.982307 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:45.982144    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:45.983525 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:45.983494    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:30:46.003445 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:46.003394    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podStartSLOduration=3.078014487 podStartE2EDuration="34.003379749s" totalImagesPullingTime="27.277747136s" totalInitContainerRuntime="3.647618126s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:30:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:30:46.001573053 +0000 UTC m=+767.010069203" watchObservedRunningTime="2026-09-05 02:30:46.003379749 +0000 UTC m=+767.011875888"
Sep 05 02:30:46.986505 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:46.986466    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:30:51.990866 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:51.990833    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:30:51.991355 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:30:51.991326    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:31:01.991970 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:31:01.991928    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:31:11.991382 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:31:11.991338    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:31:21.991731 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:31:21.991637    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:31:31.991388 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:31:31.991337    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:31:41.991682 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:31:41.991635    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:31:51.991399 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:31:51.991352    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 05 02:32:01.992457 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:01.992419    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:32:02.630549 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:02.630501    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"]
Sep 05 02:32:02.630777 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:02.630750    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container" containerID="cri-o://f667d5257ae8785bcfdc9e7ff2d2cd548083aff3d9e012db9dc81eedd0d44023" gracePeriod=30
Sep 05 02:32:02.630891 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:02.630760    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kube-rbac-proxy" containerID="cri-o://2ae20005267c070a51c3505510a7583514c8bd9f3eaf220aca78f289076e2bf2" gracePeriod=30
Sep 05 02:32:03.333253 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:03.333215    2555 generic.go:356] "Generic (PLEG): container finished" podID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerID="2ae20005267c070a51c3505510a7583514c8bd9f3eaf220aca78f289076e2bf2" exitCode=2
Sep 05 02:32:03.333622 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:03.333279    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" event={"ID":"2b9f47a6-3545-4856-85fc-7e1060f2ebe1","Type":"ContainerDied","Data":"2ae20005267c070a51c3505510a7583514c8bd9f3eaf220aca78f289076e2bf2"}
Sep 05 02:32:06.986919 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:06.986866    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 05 02:32:07.353378 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.353346    2555 generic.go:356] "Generic (PLEG): container finished" podID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerID="f667d5257ae8785bcfdc9e7ff2d2cd548083aff3d9e012db9dc81eedd0d44023" exitCode=0
Sep 05 02:32:07.353559 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.353418    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" event={"ID":"2b9f47a6-3545-4856-85fc-7e1060f2ebe1","Type":"ContainerDied","Data":"f667d5257ae8785bcfdc9e7ff2d2cd548083aff3d9e012db9dc81eedd0d44023"}
Sep 05 02:32:07.384034 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.383999    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:32:07.416704 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.416669    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-cabundle-cert\") pod \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") "
Sep 05 02:32:07.416704 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.416709    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-proxy-tls\") pod \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") "
Sep 05 02:32:07.416903 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.416735    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kserve-provision-location\") pod \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") "
Sep 05 02:32:07.416903 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.416793    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-rbac-proxy-sar-config\") pod \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") "
Sep 05 02:32:07.416903 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.416831    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll8k5\" (UniqueName: \"kubernetes.io/projected/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-api-access-ll8k5\") pod \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\" (UID: \"2b9f47a6-3545-4856-85fc-7e1060f2ebe1\") "
Sep 05 02:32:07.417091 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.417062    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b9f47a6-3545-4856-85fc-7e1060f2ebe1" (UID: "2b9f47a6-3545-4856-85fc-7e1060f2ebe1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:32:07.417163 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.417120    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2b9f47a6-3545-4856-85fc-7e1060f2ebe1" (UID: "2b9f47a6-3545-4856-85fc-7e1060f2ebe1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:07.417257 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.417217    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2b9f47a6-3545-4856-85fc-7e1060f2ebe1" (UID: "2b9f47a6-3545-4856-85fc-7e1060f2ebe1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:07.418911 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.418884    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b9f47a6-3545-4856-85fc-7e1060f2ebe1" (UID: "2b9f47a6-3545-4856-85fc-7e1060f2ebe1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:32:07.419068 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.419048    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-api-access-ll8k5" (OuterVolumeSpecName: "kube-api-access-ll8k5") pod "2b9f47a6-3545-4856-85fc-7e1060f2ebe1" (UID: "2b9f47a6-3545-4856-85fc-7e1060f2ebe1"). InnerVolumeSpecName "kube-api-access-ll8k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:32:07.518298 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.518208    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:32:07.518298 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.518241    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll8k5\" (UniqueName: \"kubernetes.io/projected/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kube-api-access-ll8k5\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:32:07.518298 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.518252    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:32:07.518298 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.518261    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:32:07.518298 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.518269    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b9f47a6-3545-4856-85fc-7e1060f2ebe1-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:32:07.795178 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.795093    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"]
Sep 05 02:32:07.795426 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.795414    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container"
Sep 05 02:32:07.795465 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.795428    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container"
Sep 05 02:32:07.795465 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.795441    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kube-rbac-proxy"
Sep 05 02:32:07.795465 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.795447    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kube-rbac-proxy"
Sep 05 02:32:07.795465 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.795464    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="storage-initializer"
Sep 05 02:32:07.795602 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.795470    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="storage-initializer"
Sep 05 02:32:07.795602 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.795522    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kube-rbac-proxy"
Sep 05 02:32:07.795602 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.795546    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" containerName="kserve-container"
Sep 05 02:32:07.798880 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.798864    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.801360 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.801343    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 05 02:32:07.801412 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.801371    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 05 02:32:07.808038 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.808012    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"]
Sep 05 02:32:07.820747 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.820714    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.820863 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.820756    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.820863 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.820789    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kthtk\" (UniqueName: \"kubernetes.io/projected/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-api-access-kthtk\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.820979 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.820902    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.821034 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.820952    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.921492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.921453    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.921691 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.921499    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.921691 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.921556    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kthtk\" (UniqueName: \"kubernetes.io/projected/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-api-access-kthtk\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.921691 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.921596    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.921691 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.921621    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.921915 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:32:07.921757    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 05 02:32:07.921915 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:32:07.921816    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-proxy-tls podName:dfd5c6f7-74c8-4311-83b7-b44472aaa6de nodeName:}" failed. No retries permitted until 2026-09-05 02:32:08.421796814 +0000 UTC m=+849.430292933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-bd89m" (UID: "dfd5c6f7-74c8-4311-83b7-b44472aaa6de") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 05 02:32:07.922060 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.922039    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.922255 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.922224    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.922292 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.922263    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:07.930485 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:07.930461    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kthtk\" (UniqueName: \"kubernetes.io/projected/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-api-access-kthtk\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:08.360511 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:08.360486    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"
Sep 05 02:32:08.360921 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:08.360516    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx" event={"ID":"2b9f47a6-3545-4856-85fc-7e1060f2ebe1","Type":"ContainerDied","Data":"d990ad09f8a40a489f62dc1c172deff9df5b424a742adf265d0a208921a3341f"}
Sep 05 02:32:08.360921 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:08.360614    2555 scope.go:117] "RemoveContainer" containerID="2ae20005267c070a51c3505510a7583514c8bd9f3eaf220aca78f289076e2bf2"
Sep 05 02:32:08.369361 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:08.369339    2555 scope.go:117] "RemoveContainer" containerID="f667d5257ae8785bcfdc9e7ff2d2cd548083aff3d9e012db9dc81eedd0d44023"
Sep 05 02:32:08.376665 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:08.376568    2555 scope.go:117] "RemoveContainer" containerID="4776ad298992920d8ca92f7430d11f63d8fd7be0565b93b75fc7b1aae37dd775"
Sep 05 02:32:08.380410 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:08.380388    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"]
Sep 05 02:32:08.382840 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:08.382821    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-bxpsx"]
Sep 05 02:32:08.427147 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:08.427114    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:08.429497 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:08.429477    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bd89m\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:08.710729 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:08.710689    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:08.989663 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:08.989639    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"]
Sep 05 02:32:08.991665 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:32:08.991637    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd5c6f7_74c8_4311_83b7_b44472aaa6de.slice/crio-9b88aa514564abdc6b6a75788280d48f9fdf564416924a05895a7fc3900728c0 WatchSource:0}: Error finding container 9b88aa514564abdc6b6a75788280d48f9fdf564416924a05895a7fc3900728c0: Status 404 returned error can't find the container with id 9b88aa514564abdc6b6a75788280d48f9fdf564416924a05895a7fc3900728c0
Sep 05 02:32:09.367196 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:09.367102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" event={"ID":"dfd5c6f7-74c8-4311-83b7-b44472aaa6de","Type":"ContainerStarted","Data":"760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824"}
Sep 05 02:32:09.367196 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:09.367182    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" event={"ID":"dfd5c6f7-74c8-4311-83b7-b44472aaa6de","Type":"ContainerStarted","Data":"9b88aa514564abdc6b6a75788280d48f9fdf564416924a05895a7fc3900728c0"}
Sep 05 02:32:09.545783 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:09.545746    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b9f47a6-3545-4856-85fc-7e1060f2ebe1" path="/var/lib/kubelet/pods/2b9f47a6-3545-4856-85fc-7e1060f2ebe1/volumes"
Sep 05 02:32:13.387375 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:13.387343    2555 generic.go:356] "Generic (PLEG): container finished" podID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerID="760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824" exitCode=0
Sep 05 02:32:13.387749 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:13.387416    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" event={"ID":"dfd5c6f7-74c8-4311-83b7-b44472aaa6de","Type":"ContainerDied","Data":"760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824"}
Sep 05 02:32:14.394234 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:14.394200    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" event={"ID":"dfd5c6f7-74c8-4311-83b7-b44472aaa6de","Type":"ContainerStarted","Data":"02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265"}
Sep 05 02:32:14.394606 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:14.394285    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" event={"ID":"dfd5c6f7-74c8-4311-83b7-b44472aaa6de","Type":"ContainerStarted","Data":"dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f"}
Sep 05 02:32:14.394606 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:14.394569    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:14.394724 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:14.394705    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:14.396124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:14.396094    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:32:14.414564 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:14.414512    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podStartSLOduration=3.486187687 podStartE2EDuration="7.414498868s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.928311181s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:32:14.411925316 +0000 UTC m=+855.420421476" watchObservedRunningTime="2026-09-05 02:32:14.414498868 +0000 UTC m=+855.422995007"
Sep 05 02:32:15.399147 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:15.399113    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:32:20.403258 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:20.403227    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:32:20.403852 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:20.403745    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:32:30.404142 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:30.404100    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:32:40.404527 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:40.404489    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:32:50.404113 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:50.404071    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:32:59.476308 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:59.476237    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:32:59.476308 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:32:59.476237    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:33:00.403924 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:00.403887    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:33:07.723547 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.723511    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"]
Sep 05 02:33:07.728237 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.728218    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:07.730632 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.730610    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 05 02:33:07.730717 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.730685    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 05 02:33:07.735928 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.735906    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"]
Sep 05 02:33:07.832763 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.832723    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/296c88b5-cf21-426d-803c-d50e658052e9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2tlsm\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:07.832933 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.832794    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/296c88b5-cf21-426d-803c-d50e658052e9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2tlsm\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:07.832933 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.832896    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqdt\" (UniqueName: \"kubernetes.io/projected/296c88b5-cf21-426d-803c-d50e658052e9-kube-api-access-ztqdt\") pod \"message-dumper-predictor-6575bcfdfd-2tlsm\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:07.934000 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.933964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/296c88b5-cf21-426d-803c-d50e658052e9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2tlsm\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:07.934197 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.934011    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/296c88b5-cf21-426d-803c-d50e658052e9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2tlsm\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:07.934197 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.934060    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqdt\" (UniqueName: \"kubernetes.io/projected/296c88b5-cf21-426d-803c-d50e658052e9-kube-api-access-ztqdt\") pod \"message-dumper-predictor-6575bcfdfd-2tlsm\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:07.934197 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:33:07.934141    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 05 02:33:07.934345 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:33:07.934211    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/296c88b5-cf21-426d-803c-d50e658052e9-proxy-tls podName:296c88b5-cf21-426d-803c-d50e658052e9 nodeName:}" failed. No retries permitted until 2026-09-05 02:33:08.434190752 +0000 UTC m=+909.442686882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/296c88b5-cf21-426d-803c-d50e658052e9-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-2tlsm" (UID: "296c88b5-cf21-426d-803c-d50e658052e9") : secret "message-dumper-predictor-serving-cert" not found
Sep 05 02:33:07.934786 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.934767    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/296c88b5-cf21-426d-803c-d50e658052e9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-2tlsm\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:07.944660 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:07.944638    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqdt\" (UniqueName: \"kubernetes.io/projected/296c88b5-cf21-426d-803c-d50e658052e9-kube-api-access-ztqdt\") pod \"message-dumper-predictor-6575bcfdfd-2tlsm\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:08.437988 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:08.437954    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/296c88b5-cf21-426d-803c-d50e658052e9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2tlsm\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:08.440404 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:08.440378    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/296c88b5-cf21-426d-803c-d50e658052e9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-2tlsm\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:08.639180 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:08.639145    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:08.910793 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:08.910733    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"]
Sep 05 02:33:08.913165 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:33:08.913137    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296c88b5_cf21_426d_803c_d50e658052e9.slice/crio-f3b25b6bda446430fcd8c0fdacc62fef03c6da809d2d8d8b3610c7a9cc734a69 WatchSource:0}: Error finding container f3b25b6bda446430fcd8c0fdacc62fef03c6da809d2d8d8b3610c7a9cc734a69: Status 404 returned error can't find the container with id f3b25b6bda446430fcd8c0fdacc62fef03c6da809d2d8d8b3610c7a9cc734a69
Sep 05 02:33:09.641795 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:09.641755    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm" event={"ID":"296c88b5-cf21-426d-803c-d50e658052e9","Type":"ContainerStarted","Data":"f3b25b6bda446430fcd8c0fdacc62fef03c6da809d2d8d8b3610c7a9cc734a69"}
Sep 05 02:33:10.404353 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:10.404313    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:33:10.648168 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:10.648138    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm" event={"ID":"296c88b5-cf21-426d-803c-d50e658052e9","Type":"ContainerStarted","Data":"1c96e7324e5120004f4c58697d41b47172cadf74e4974cca9c43be43c5e72ea1"}
Sep 05 02:33:10.648345 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:10.648193    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm" event={"ID":"296c88b5-cf21-426d-803c-d50e658052e9","Type":"ContainerStarted","Data":"8646d981d51262d603281fe82f8d1b6c1f25b00472ab3537f716fbdcf0e4526a"}
Sep 05 02:33:10.648345 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:10.648302    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:10.668177 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:10.668127    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm" podStartSLOduration=2.398645115 podStartE2EDuration="3.668110181s" totalImagesPullingTime="1.269465066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:33:10.667167774 +0000 UTC m=+911.675663934" watchObservedRunningTime="2026-09-05 02:33:10.668110181 +0000 UTC m=+911.676606392"
Sep 05 02:33:11.652637 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:11.652604    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:11.654659 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:11.654631    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:18.666765 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:18.666738    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:33:20.404279 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:20.404240    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:33:30.405453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:30.405422    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:33:37.864565 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:37.864503    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"]
Sep 05 02:33:37.864925 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:37.864865    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container" containerID="cri-o://dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f" gracePeriod=30
Sep 05 02:33:37.864983 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:37.864896    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kube-rbac-proxy" containerID="cri-o://02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265" gracePeriod=30
Sep 05 02:33:38.774596 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:38.774564    2555 generic.go:356] "Generic (PLEG): container finished" podID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerID="02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265" exitCode=2
Sep 05 02:33:38.774775 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:38.774645    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" event={"ID":"dfd5c6f7-74c8-4311-83b7-b44472aaa6de","Type":"ContainerDied","Data":"02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265"}
Sep 05 02:33:40.399985 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:40.399944    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 05 02:33:40.404254 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:40.404224    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 05 02:33:42.606494 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.606465    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:33:42.724845 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.724757    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-rbac-proxy-sar-config\") pod \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") "
Sep 05 02:33:42.724845 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.724797    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kserve-provision-location\") pod \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") "
Sep 05 02:33:42.724845 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.724848    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-cabundle-cert\") pod \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") "
Sep 05 02:33:42.725125 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.724946    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-proxy-tls\") pod \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") "
Sep 05 02:33:42.725125 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.725012    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kthtk\" (UniqueName: \"kubernetes.io/projected/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-api-access-kthtk\") pod \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\" (UID: \"dfd5c6f7-74c8-4311-83b7-b44472aaa6de\") "
Sep 05 02:33:42.725221 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.725149    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dfd5c6f7-74c8-4311-83b7-b44472aaa6de" (UID: "dfd5c6f7-74c8-4311-83b7-b44472aaa6de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:33:42.725221 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.725178    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dfd5c6f7-74c8-4311-83b7-b44472aaa6de" (UID: "dfd5c6f7-74c8-4311-83b7-b44472aaa6de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:42.725221 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.725187    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dfd5c6f7-74c8-4311-83b7-b44472aaa6de" (UID: "dfd5c6f7-74c8-4311-83b7-b44472aaa6de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:42.725349 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.725297    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:33:42.725349 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.725312    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:33:42.725349 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.725322    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:33:42.727051 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.727029    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dfd5c6f7-74c8-4311-83b7-b44472aaa6de" (UID: "dfd5c6f7-74c8-4311-83b7-b44472aaa6de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:33:42.727161 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.727126    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-api-access-kthtk" (OuterVolumeSpecName: "kube-api-access-kthtk") pod "dfd5c6f7-74c8-4311-83b7-b44472aaa6de" (UID: "dfd5c6f7-74c8-4311-83b7-b44472aaa6de"). InnerVolumeSpecName "kube-api-access-kthtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:33:42.794862 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.794831    2555 generic.go:356] "Generic (PLEG): container finished" podID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerID="dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f" exitCode=0
Sep 05 02:33:42.795010 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.794909    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"
Sep 05 02:33:42.795010 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.794916    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" event={"ID":"dfd5c6f7-74c8-4311-83b7-b44472aaa6de","Type":"ContainerDied","Data":"dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f"}
Sep 05 02:33:42.795010 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.794979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m" event={"ID":"dfd5c6f7-74c8-4311-83b7-b44472aaa6de","Type":"ContainerDied","Data":"9b88aa514564abdc6b6a75788280d48f9fdf564416924a05895a7fc3900728c0"}
Sep 05 02:33:42.795010 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.795008    2555 scope.go:117] "RemoveContainer" containerID="02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265"
Sep 05 02:33:42.803908 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.803893    2555 scope.go:117] "RemoveContainer" containerID="dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f"
Sep 05 02:33:42.811191 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.811172    2555 scope.go:117] "RemoveContainer" containerID="760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824"
Sep 05 02:33:42.816123 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.816102    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"]
Sep 05 02:33:42.818852 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.818835    2555 scope.go:117] "RemoveContainer" containerID="02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265"
Sep 05 02:33:42.819108 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:33:42.819080    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265\": container with ID starting with 02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265 not found: ID does not exist" containerID="02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265"
Sep 05 02:33:42.819158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.819117    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265"} err="failed to get container status \"02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265\": rpc error: code = NotFound desc = could not find container \"02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265\": container with ID starting with 02efc76bfc0a4bca35016478b8c87d8bcad9d607b149da4b269b0e25299d8265 not found: ID does not exist"
Sep 05 02:33:42.819158 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.819134    2555 scope.go:117] "RemoveContainer" containerID="dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f"
Sep 05 02:33:42.819427 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:33:42.819410    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f\": container with ID starting with dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f not found: ID does not exist" containerID="dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f"
Sep 05 02:33:42.819467 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.819434    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f"} err="failed to get container status \"dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f\": rpc error: code = NotFound desc = could not find container \"dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f\": container with ID starting with dc33740239bf9bac0267e0ff0d70b7aaffb536b7d5b16ada42c0c281d1e5cc4f not found: ID does not exist"
Sep 05 02:33:42.819467 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.819449    2555 scope.go:117] "RemoveContainer" containerID="760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824"
Sep 05 02:33:42.819691 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:33:42.819674    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824\": container with ID starting with 760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824 not found: ID does not exist" containerID="760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824"
Sep 05 02:33:42.819736 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.819697    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824"} err="failed to get container status \"760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824\": rpc error: code = NotFound desc = could not find container \"760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824\": container with ID starting with 760a37f2808d0c742ff302bd7b71bc3bdb7452ccef5b1565540121e1b2b0f824 not found: ID does not exist"
Sep 05 02:33:42.820215 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.820198    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bd89m"]
Sep 05 02:33:42.826102 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.826083    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:33:42.826157 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:42.826103    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kthtk\" (UniqueName: \"kubernetes.io/projected/dfd5c6f7-74c8-4311-83b7-b44472aaa6de-kube-api-access-kthtk\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:33:43.041733 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.041646    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"]
Sep 05 02:33:43.042075 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.042056    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="storage-initializer"
Sep 05 02:33:43.042163 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.042079    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="storage-initializer"
Sep 05 02:33:43.042163 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.042101    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container"
Sep 05 02:33:43.042163 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.042109    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container"
Sep 05 02:33:43.042163 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.042125    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kube-rbac-proxy"
Sep 05 02:33:43.042163 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.042134    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kube-rbac-proxy"
Sep 05 02:33:43.042849 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.042212    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kserve-container"
Sep 05 02:33:43.042849 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.042225    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" containerName="kube-rbac-proxy"
Sep 05 02:33:43.046853 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.046831    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.048951 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.048928    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 05 02:33:43.049068 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.048968    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:33:43.049068 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.049060    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:33:43.049240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.049223    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 05 02:33:43.054557 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.054523    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"]
Sep 05 02:33:43.127819 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.127786    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.127994 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.127838    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfd6fdd8-0395-4792-ba6a-89f835336daa-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.127994 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.127872    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfd6fdd8-0395-4792-ba6a-89f835336daa-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.127994 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.127943    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwgrk\" (UniqueName: \"kubernetes.io/projected/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-api-access-dwgrk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.127994 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.127960    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.228682 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.228643    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.228900 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.228689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfd6fdd8-0395-4792-ba6a-89f835336daa-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.228900 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.228727    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfd6fdd8-0395-4792-ba6a-89f835336daa-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.228900 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.228791    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwgrk\" (UniqueName: \"kubernetes.io/projected/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-api-access-dwgrk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.228900 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.228814    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.229205 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.229182    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfd6fdd8-0395-4792-ba6a-89f835336daa-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.229423 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.229401    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.229482 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.229418    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.231091 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.231075    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfd6fdd8-0395-4792-ba6a-89f835336daa-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.236810 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.236792    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwgrk\" (UniqueName: \"kubernetes.io/projected/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-api-access-dwgrk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.358552 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.358446    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:43.544101 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.544065    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfd5c6f7-74c8-4311-83b7-b44472aaa6de" path="/var/lib/kubelet/pods/dfd5c6f7-74c8-4311-83b7-b44472aaa6de/volumes"
Sep 05 02:33:43.629091 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.629066    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"]
Sep 05 02:33:43.631422 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:33:43.631391    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd6fdd8_0395_4792_ba6a_89f835336daa.slice/crio-8bc4e1be76da34cfa93a9587a5f961e5c0e72f32832f837925666cd42decc95e WatchSource:0}: Error finding container 8bc4e1be76da34cfa93a9587a5f961e5c0e72f32832f837925666cd42decc95e: Status 404 returned error can't find the container with id 8bc4e1be76da34cfa93a9587a5f961e5c0e72f32832f837925666cd42decc95e
Sep 05 02:33:43.802683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.802643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" event={"ID":"bfd6fdd8-0395-4792-ba6a-89f835336daa","Type":"ContainerStarted","Data":"aee7a8fb8424e563c756ea88f36de6db56f7d47fad41662317d95b36b1b71a28"}
Sep 05 02:33:43.802850 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:43.802706    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" event={"ID":"bfd6fdd8-0395-4792-ba6a-89f835336daa","Type":"ContainerStarted","Data":"8bc4e1be76da34cfa93a9587a5f961e5c0e72f32832f837925666cd42decc95e"}
Sep 05 02:33:47.822258 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:47.822221    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerID="aee7a8fb8424e563c756ea88f36de6db56f7d47fad41662317d95b36b1b71a28" exitCode=0
Sep 05 02:33:47.822659 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:47.822291    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" event={"ID":"bfd6fdd8-0395-4792-ba6a-89f835336daa","Type":"ContainerDied","Data":"aee7a8fb8424e563c756ea88f36de6db56f7d47fad41662317d95b36b1b71a28"}
Sep 05 02:33:48.829077 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:48.829042    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" event={"ID":"bfd6fdd8-0395-4792-ba6a-89f835336daa","Type":"ContainerStarted","Data":"d9199981a309b70ca3d9e2d95e8a2ca8ecb292411bb15e73a05f9cd7f07b94a6"}
Sep 05 02:33:48.829477 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:48.829102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" event={"ID":"bfd6fdd8-0395-4792-ba6a-89f835336daa","Type":"ContainerStarted","Data":"49816313645a05b052aedbd56afc459a84cb659055099c13190f0446e3ae3483"}
Sep 05 02:33:48.829477 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:48.829400    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:48.829578 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:48.829546    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:48.831085 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:48.831056    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 05 02:33:48.847272 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:48.847230    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podStartSLOduration=1.952591996 podStartE2EDuration="5.847215948s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.894623952s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:33:48.845836979 +0000 UTC m=+949.854333141" watchObservedRunningTime="2026-09-05 02:33:48.847215948 +0000 UTC m=+949.855712086"
Sep 05 02:33:49.834332 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:49.834289    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 05 02:33:54.839253 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:54.839220    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:33:54.839785 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:33:54.839755    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 05 02:34:04.840415 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:34:04.840373    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 05 02:34:14.840342 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:34:14.840305    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 05 02:34:24.839921 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:34:24.839836    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 05 02:34:34.839909 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:34:34.839866    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 05 02:34:44.839995 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:34:44.839953    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 05 02:34:54.840338 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:34:54.840293    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 05 02:35:02.774103 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:02.774071    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-2tlsm_296c88b5-cf21-426d-803c-d50e658052e9/kserve-container/0.log"
Sep 05 02:35:02.949965 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:02.949932    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"]
Sep 05 02:35:02.950238 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:02.950211    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm" podUID="296c88b5-cf21-426d-803c-d50e658052e9" containerName="kserve-container" containerID="cri-o://8646d981d51262d603281fe82f8d1b6c1f25b00472ab3537f716fbdcf0e4526a" gracePeriod=30
Sep 05 02:35:02.950325 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:02.950249    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm" podUID="296c88b5-cf21-426d-803c-d50e658052e9" containerName="kube-rbac-proxy" containerID="cri-o://1c96e7324e5120004f4c58697d41b47172cadf74e4974cca9c43be43c5e72ea1" gracePeriod=30
Sep 05 02:35:03.138121 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.138089    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"]
Sep 05 02:35:03.141799 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.141777    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.146852 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.146678    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 05 02:35:03.146852 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.146708    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 05 02:35:03.153291 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.153206    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"]
Sep 05 02:35:03.159267 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.159234    2555 generic.go:356] "Generic (PLEG): container finished" podID="296c88b5-cf21-426d-803c-d50e658052e9" containerID="1c96e7324e5120004f4c58697d41b47172cadf74e4974cca9c43be43c5e72ea1" exitCode=2
Sep 05 02:35:03.159267 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.159266    2555 generic.go:356] "Generic (PLEG): container finished" podID="296c88b5-cf21-426d-803c-d50e658052e9" containerID="8646d981d51262d603281fe82f8d1b6c1f25b00472ab3537f716fbdcf0e4526a" exitCode=2
Sep 05 02:35:03.159449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.159277    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm" event={"ID":"296c88b5-cf21-426d-803c-d50e658052e9","Type":"ContainerDied","Data":"1c96e7324e5120004f4c58697d41b47172cadf74e4974cca9c43be43c5e72ea1"}
Sep 05 02:35:03.159449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.159352    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm" event={"ID":"296c88b5-cf21-426d-803c-d50e658052e9","Type":"ContainerDied","Data":"8646d981d51262d603281fe82f8d1b6c1f25b00472ab3537f716fbdcf0e4526a"}
Sep 05 02:35:03.198633 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.198608    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:35:03.210816 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.210788    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/296c88b5-cf21-426d-803c-d50e658052e9-kube-rbac-proxy-sar-config\") pod \"296c88b5-cf21-426d-803c-d50e658052e9\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") "
Sep 05 02:35:03.210893 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.210833    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztqdt\" (UniqueName: \"kubernetes.io/projected/296c88b5-cf21-426d-803c-d50e658052e9-kube-api-access-ztqdt\") pod \"296c88b5-cf21-426d-803c-d50e658052e9\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") "
Sep 05 02:35:03.210893 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.210863    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/296c88b5-cf21-426d-803c-d50e658052e9-proxy-tls\") pod \"296c88b5-cf21-426d-803c-d50e658052e9\" (UID: \"296c88b5-cf21-426d-803c-d50e658052e9\") "
Sep 05 02:35:03.210982 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.210970    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6466c540-7eef-459e-99bd-f67a1beb1fdd-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.211017 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.210988    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.211065 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.211018    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.211106 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.211072    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6466c540-7eef-459e-99bd-f67a1beb1fdd-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.211150 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.211101    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5knx\" (UniqueName: \"kubernetes.io/projected/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-api-access-c5knx\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.211187 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.211150    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296c88b5-cf21-426d-803c-d50e658052e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "296c88b5-cf21-426d-803c-d50e658052e9" (UID: "296c88b5-cf21-426d-803c-d50e658052e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:03.212856 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.212825    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296c88b5-cf21-426d-803c-d50e658052e9-kube-api-access-ztqdt" (OuterVolumeSpecName: "kube-api-access-ztqdt") pod "296c88b5-cf21-426d-803c-d50e658052e9" (UID: "296c88b5-cf21-426d-803c-d50e658052e9"). InnerVolumeSpecName "kube-api-access-ztqdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:35:03.212944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.212872    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/296c88b5-cf21-426d-803c-d50e658052e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "296c88b5-cf21-426d-803c-d50e658052e9" (UID: "296c88b5-cf21-426d-803c-d50e658052e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:35:03.311719 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.311679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6466c540-7eef-459e-99bd-f67a1beb1fdd-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.311719 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.311715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5knx\" (UniqueName: \"kubernetes.io/projected/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-api-access-c5knx\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.311944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.311769    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6466c540-7eef-459e-99bd-f67a1beb1fdd-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.311944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.311785    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.311944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.311812    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.311944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.311850    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/296c88b5-cf21-426d-803c-d50e658052e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:35:03.311944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.311860    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqdt\" (UniqueName: \"kubernetes.io/projected/296c88b5-cf21-426d-803c-d50e658052e9-kube-api-access-ztqdt\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:35:03.311944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.311869    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/296c88b5-cf21-426d-803c-d50e658052e9-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:35:03.311944 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:35:03.311918    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 05 02:35:03.312261 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:35:03.311988    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6466c540-7eef-459e-99bd-f67a1beb1fdd-proxy-tls podName:6466c540-7eef-459e-99bd-f67a1beb1fdd nodeName:}" failed. No retries permitted until 2026-09-05 02:35:03.811968066 +0000 UTC m=+1024.820464202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6466c540-7eef-459e-99bd-f67a1beb1fdd-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-4smct" (UID: "6466c540-7eef-459e-99bd-f67a1beb1fdd") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 05 02:35:03.312261 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.312147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6466c540-7eef-459e-99bd-f67a1beb1fdd-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.312429 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.312412    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.312465 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.312413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.320338 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.320287    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5knx\" (UniqueName: \"kubernetes.io/projected/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-api-access-c5knx\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.815854 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.815814    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6466c540-7eef-459e-99bd-f67a1beb1fdd-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:03.818151 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:03.818131    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6466c540-7eef-459e-99bd-f67a1beb1fdd-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-4smct\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:04.053282 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:04.053228    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:35:04.165785 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:04.165751    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm" event={"ID":"296c88b5-cf21-426d-803c-d50e658052e9","Type":"ContainerDied","Data":"f3b25b6bda446430fcd8c0fdacc62fef03c6da809d2d8d8b3610c7a9cc734a69"}
Sep 05 02:35:04.165976 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:04.165817    2555 scope.go:117] "RemoveContainer" containerID="1c96e7324e5120004f4c58697d41b47172cadf74e4974cca9c43be43c5e72ea1"
Sep 05 02:35:04.165976 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:04.165765    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"
Sep 05 02:35:04.173719 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:04.173697    2555 scope.go:117] "RemoveContainer" containerID="8646d981d51262d603281fe82f8d1b6c1f25b00472ab3537f716fbdcf0e4526a"
Sep 05 02:35:04.183134 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:04.183112    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"]
Sep 05 02:35:04.186499 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:04.186476    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-2tlsm"]
Sep 05 02:35:04.328034 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:04.328009    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"]
Sep 05 02:35:04.330146 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:35:04.330124    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6466c540_7eef_459e_99bd_f67a1beb1fdd.slice/crio-522d6590e5f853157dd003387d120711af8b8eda3addb4e476aa662f17ac8632 WatchSource:0}: Error finding container 522d6590e5f853157dd003387d120711af8b8eda3addb4e476aa662f17ac8632: Status 404 returned error can't find the container with id 522d6590e5f853157dd003387d120711af8b8eda3addb4e476aa662f17ac8632
Sep 05 02:35:04.539706 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:04.539679    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:35:05.171598 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:05.171556    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" event={"ID":"6466c540-7eef-459e-99bd-f67a1beb1fdd","Type":"ContainerStarted","Data":"1ebfb0cab4b521759441ba5e001eb7f8e089996d98fda0d9119bcc80bb7a3768"}
Sep 05 02:35:05.172061 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:05.171636    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" event={"ID":"6466c540-7eef-459e-99bd-f67a1beb1fdd","Type":"ContainerStarted","Data":"522d6590e5f853157dd003387d120711af8b8eda3addb4e476aa662f17ac8632"}
Sep 05 02:35:05.544028 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:05.543946    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="296c88b5-cf21-426d-803c-d50e658052e9" path="/var/lib/kubelet/pods/296c88b5-cf21-426d-803c-d50e658052e9/volumes"
Sep 05 02:35:13.115356 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:13.115320    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"]
Sep 05 02:35:13.115805 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:13.115561    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container" containerID="cri-o://49816313645a05b052aedbd56afc459a84cb659055099c13190f0446e3ae3483" gracePeriod=30
Sep 05 02:35:13.115805 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:13.115602    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kube-rbac-proxy" containerID="cri-o://d9199981a309b70ca3d9e2d95e8a2ca8ecb292411bb15e73a05f9cd7f07b94a6" gracePeriod=30
Sep 05 02:35:14.217064 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:14.217025    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerID="d9199981a309b70ca3d9e2d95e8a2ca8ecb292411bb15e73a05f9cd7f07b94a6" exitCode=2
Sep 05 02:35:14.217425 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:14.217070    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" event={"ID":"bfd6fdd8-0395-4792-ba6a-89f835336daa","Type":"ContainerDied","Data":"d9199981a309b70ca3d9e2d95e8a2ca8ecb292411bb15e73a05f9cd7f07b94a6"}
Sep 05 02:35:14.538995 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:14.538900    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 05 02:35:14.835545 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:14.835440    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 05 02:35:18.238946 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.238907    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerID="49816313645a05b052aedbd56afc459a84cb659055099c13190f0446e3ae3483" exitCode=0
Sep 05 02:35:18.239340 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.238977    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" event={"ID":"bfd6fdd8-0395-4792-ba6a-89f835336daa","Type":"ContainerDied","Data":"49816313645a05b052aedbd56afc459a84cb659055099c13190f0446e3ae3483"}
Sep 05 02:35:18.361792 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.361769    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:35:18.440271 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.440239    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfd6fdd8-0395-4792-ba6a-89f835336daa-proxy-tls\") pod \"bfd6fdd8-0395-4792-ba6a-89f835336daa\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") "
Sep 05 02:35:18.440435 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.440283    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-rbac-proxy-sar-config\") pod \"bfd6fdd8-0395-4792-ba6a-89f835336daa\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") "
Sep 05 02:35:18.440435 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.440315    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwgrk\" (UniqueName: \"kubernetes.io/projected/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-api-access-dwgrk\") pod \"bfd6fdd8-0395-4792-ba6a-89f835336daa\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") "
Sep 05 02:35:18.440435 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.440379    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-cabundle-cert\") pod \"bfd6fdd8-0395-4792-ba6a-89f835336daa\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") "
Sep 05 02:35:18.440435 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.440408    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfd6fdd8-0395-4792-ba6a-89f835336daa-kserve-provision-location\") pod \"bfd6fdd8-0395-4792-ba6a-89f835336daa\" (UID: \"bfd6fdd8-0395-4792-ba6a-89f835336daa\") "
Sep 05 02:35:18.440797 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.440769    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bfd6fdd8-0395-4792-ba6a-89f835336daa" (UID: "bfd6fdd8-0395-4792-ba6a-89f835336daa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:18.440860 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.440795    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd6fdd8-0395-4792-ba6a-89f835336daa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bfd6fdd8-0395-4792-ba6a-89f835336daa" (UID: "bfd6fdd8-0395-4792-ba6a-89f835336daa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:35:18.440860 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.440771    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bfd6fdd8-0395-4792-ba6a-89f835336daa" (UID: "bfd6fdd8-0395-4792-ba6a-89f835336daa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:18.442440 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.442416    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-api-access-dwgrk" (OuterVolumeSpecName: "kube-api-access-dwgrk") pod "bfd6fdd8-0395-4792-ba6a-89f835336daa" (UID: "bfd6fdd8-0395-4792-ba6a-89f835336daa"). InnerVolumeSpecName "kube-api-access-dwgrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:35:18.442547 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.442436    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfd6fdd8-0395-4792-ba6a-89f835336daa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bfd6fdd8-0395-4792-ba6a-89f835336daa" (UID: "bfd6fdd8-0395-4792-ba6a-89f835336daa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:35:18.541997 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.541908    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwgrk\" (UniqueName: \"kubernetes.io/projected/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-api-access-dwgrk\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:35:18.541997 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.541941    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:35:18.541997 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.541952    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfd6fdd8-0395-4792-ba6a-89f835336daa-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:35:18.541997 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.541964    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfd6fdd8-0395-4792-ba6a-89f835336daa-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:35:18.541997 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:18.541972    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfd6fdd8-0395-4792-ba6a-89f835336daa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:35:19.245029 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:19.244990    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf" event={"ID":"bfd6fdd8-0395-4792-ba6a-89f835336daa","Type":"ContainerDied","Data":"8bc4e1be76da34cfa93a9587a5f961e5c0e72f32832f837925666cd42decc95e"}
Sep 05 02:35:19.245502 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:19.245057    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"
Sep 05 02:35:19.245502 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:19.245081    2555 scope.go:117] "RemoveContainer" containerID="d9199981a309b70ca3d9e2d95e8a2ca8ecb292411bb15e73a05f9cd7f07b94a6"
Sep 05 02:35:19.253330 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:19.253143    2555 scope.go:117] "RemoveContainer" containerID="49816313645a05b052aedbd56afc459a84cb659055099c13190f0446e3ae3483"
Sep 05 02:35:19.260549 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:19.260516    2555 scope.go:117] "RemoveContainer" containerID="aee7a8fb8424e563c756ea88f36de6db56f7d47fad41662317d95b36b1b71a28"
Sep 05 02:35:19.267449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:19.267427    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"]
Sep 05 02:35:19.271243 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:19.271219    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-snxwf"]
Sep 05 02:35:19.544362 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:19.544286    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" path="/var/lib/kubelet/pods/bfd6fdd8-0395-4792-ba6a-89f835336daa/volumes"
Sep 05 02:35:34.313907 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:34.313871    2555 generic.go:356] "Generic (PLEG): container finished" podID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerID="1ebfb0cab4b521759441ba5e001eb7f8e089996d98fda0d9119bcc80bb7a3768" exitCode=0
Sep 05 02:35:34.314278 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:34.313944    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" event={"ID":"6466c540-7eef-459e-99bd-f67a1beb1fdd","Type":"ContainerDied","Data":"1ebfb0cab4b521759441ba5e001eb7f8e089996d98fda0d9119bcc80bb7a3768"}
Sep 05 02:35:34.315178 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:35:34.315164    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:37:33.931624 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:33.931581    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" event={"ID":"6466c540-7eef-459e-99bd-f67a1beb1fdd","Type":"ContainerStarted","Data":"cdd995203779303de7f9cd9888a85b445f700636de1c8f8e96372a8ca274b03b"}
Sep 05 02:37:33.932061 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:33.931656    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" event={"ID":"6466c540-7eef-459e-99bd-f67a1beb1fdd","Type":"ContainerStarted","Data":"5aaeadd7568ef9aa59439b573f7955a116459951e5f6d760622fc4be59e3f9e9"}
Sep 05 02:37:33.932061 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:33.931690    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:37:33.951787 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:33.951716    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" podStartSLOduration=2.940829501 podStartE2EDuration="2m30.951696029s" totalImagesPullingTime="1m58.579973028s" totalInitContainerRuntime="29.4308935s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:37:33.951278123 +0000 UTC m=+1174.959774289" watchObservedRunningTime="2026-09-05 02:37:33.951696029 +0000 UTC m=+1174.960192168"
Sep 05 02:37:34.936331 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:34.936299    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:37:34.937995 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:34.937957    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 05 02:37:35.941305 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:35.941257    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 05 02:37:40.945960 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:40.945932    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:37:40.946492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:40.946462    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 05 02:37:50.947737 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:50.947704    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:37:59.442634 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:59.442604    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"]
Sep 05 02:37:59.443008 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:59.442885    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kserve-container" containerID="cri-o://5aaeadd7568ef9aa59439b573f7955a116459951e5f6d760622fc4be59e3f9e9" gracePeriod=30
Sep 05 02:37:59.443076 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:59.443000    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kube-rbac-proxy" containerID="cri-o://cdd995203779303de7f9cd9888a85b445f700636de1c8f8e96372a8ca274b03b" gracePeriod=30
Sep 05 02:37:59.512492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:59.512460    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:37:59.512698 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:37:59.512465    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:38:00.054833 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:00.054798    2555 generic.go:356] "Generic (PLEG): container finished" podID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerID="cdd995203779303de7f9cd9888a85b445f700636de1c8f8e96372a8ca274b03b" exitCode=2
Sep 05 02:38:00.054995 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:00.054859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" event={"ID":"6466c540-7eef-459e-99bd-f67a1beb1fdd","Type":"ContainerDied","Data":"cdd995203779303de7f9cd9888a85b445f700636de1c8f8e96372a8ca274b03b"}
Sep 05 02:38:00.941818 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:00.941775    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 05 02:38:00.947077 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:00.947045    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 05 02:38:04.074196 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.074159    2555 generic.go:356] "Generic (PLEG): container finished" podID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerID="5aaeadd7568ef9aa59439b573f7955a116459951e5f6d760622fc4be59e3f9e9" exitCode=0
Sep 05 02:38:04.074488 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.074172    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" event={"ID":"6466c540-7eef-459e-99bd-f67a1beb1fdd","Type":"ContainerDied","Data":"5aaeadd7568ef9aa59439b573f7955a116459951e5f6d760622fc4be59e3f9e9"}
Sep 05 02:38:04.214694 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.214626    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:38:04.336672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.336644    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5knx\" (UniqueName: \"kubernetes.io/projected/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-api-access-c5knx\") pod \"6466c540-7eef-459e-99bd-f67a1beb1fdd\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") "
Sep 05 02:38:04.336672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.336683    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6466c540-7eef-459e-99bd-f67a1beb1fdd-kserve-provision-location\") pod \"6466c540-7eef-459e-99bd-f67a1beb1fdd\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") "
Sep 05 02:38:04.336926 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.336728    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-cabundle-cert\") pod \"6466c540-7eef-459e-99bd-f67a1beb1fdd\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") "
Sep 05 02:38:04.336926 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.336773    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-rbac-proxy-sar-config\") pod \"6466c540-7eef-459e-99bd-f67a1beb1fdd\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") "
Sep 05 02:38:04.336926 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.336840    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6466c540-7eef-459e-99bd-f67a1beb1fdd-proxy-tls\") pod \"6466c540-7eef-459e-99bd-f67a1beb1fdd\" (UID: \"6466c540-7eef-459e-99bd-f67a1beb1fdd\") "
Sep 05 02:38:04.337135 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.337103    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6466c540-7eef-459e-99bd-f67a1beb1fdd" (UID: "6466c540-7eef-459e-99bd-f67a1beb1fdd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:38:04.337466 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.337437    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6466c540-7eef-459e-99bd-f67a1beb1fdd" (UID: "6466c540-7eef-459e-99bd-f67a1beb1fdd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:38:04.338987 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.338950    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6466c540-7eef-459e-99bd-f67a1beb1fdd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6466c540-7eef-459e-99bd-f67a1beb1fdd" (UID: "6466c540-7eef-459e-99bd-f67a1beb1fdd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:38:04.339259 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.339236    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-api-access-c5knx" (OuterVolumeSpecName: "kube-api-access-c5knx") pod "6466c540-7eef-459e-99bd-f67a1beb1fdd" (UID: "6466c540-7eef-459e-99bd-f67a1beb1fdd"). InnerVolumeSpecName "kube-api-access-c5knx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:38:04.340775 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.340750    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6466c540-7eef-459e-99bd-f67a1beb1fdd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6466c540-7eef-459e-99bd-f67a1beb1fdd" (UID: "6466c540-7eef-459e-99bd-f67a1beb1fdd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:38:04.437972 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.437938    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:38:04.437972 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.437970    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6466c540-7eef-459e-99bd-f67a1beb1fdd-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:38:04.437972 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.437979    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5knx\" (UniqueName: \"kubernetes.io/projected/6466c540-7eef-459e-99bd-f67a1beb1fdd-kube-api-access-c5knx\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:38:04.438174 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.437987    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6466c540-7eef-459e-99bd-f67a1beb1fdd-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:38:04.438174 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.437996    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6466c540-7eef-459e-99bd-f67a1beb1fdd-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:38:04.730144 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730105    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"]
Sep 05 02:38:04.730444 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730432    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="storage-initializer"
Sep 05 02:38:04.730444 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730446    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="storage-initializer"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730456    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="storage-initializer"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730461    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="storage-initializer"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730467    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="296c88b5-cf21-426d-803c-d50e658052e9" containerName="kube-rbac-proxy"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730473    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="296c88b5-cf21-426d-803c-d50e658052e9" containerName="kube-rbac-proxy"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730486    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kserve-container"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730491    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kserve-container"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730498    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kube-rbac-proxy"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730504    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kube-rbac-proxy"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730511    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730516    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730524    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kube-rbac-proxy"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730529    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kube-rbac-proxy"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730551    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="296c88b5-cf21-426d-803c-d50e658052e9" containerName="kserve-container"
Sep 05 02:38:04.730563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730557    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="296c88b5-cf21-426d-803c-d50e658052e9" containerName="kserve-container"
Sep 05 02:38:04.731022 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730606    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kserve-container"
Sep 05 02:38:04.731022 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730615    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="296c88b5-cf21-426d-803c-d50e658052e9" containerName="kube-rbac-proxy"
Sep 05 02:38:04.731022 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730621    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kserve-container"
Sep 05 02:38:04.731022 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730627    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" containerName="kube-rbac-proxy"
Sep 05 02:38:04.731022 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730632    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="296c88b5-cf21-426d-803c-d50e658052e9" containerName="kserve-container"
Sep 05 02:38:04.731022 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.730638    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd6fdd8-0395-4792-ba6a-89f835336daa" containerName="kube-rbac-proxy"
Sep 05 02:38:04.772891 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.772858    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"]
Sep 05 02:38:04.773049 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.773004    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:04.776824 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.776805    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 05 02:38:04.776947 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.776852    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 05 02:38:04.941347 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.941308    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:04.941347 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.941356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:04.941577 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.941431    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8428b\" (UniqueName: \"kubernetes.io/projected/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-api-access-8428b\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:04.941577 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.941490    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:04.941577 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:04.941555    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.041950 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.041872    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8428b\" (UniqueName: \"kubernetes.io/projected/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-api-access-8428b\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.041950 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.041921    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.042152 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.041961    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.042152 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.041982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.042152 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.042008    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.042562 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.042514    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.042761 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.042741    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.042820 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.042750    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.044428 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.044406    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.056033 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.056009    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8428b\" (UniqueName: \"kubernetes.io/projected/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-api-access-8428b\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bg7sm\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.081007 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.080978    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct" event={"ID":"6466c540-7eef-459e-99bd-f67a1beb1fdd","Type":"ContainerDied","Data":"522d6590e5f853157dd003387d120711af8b8eda3addb4e476aa662f17ac8632"}
Sep 05 02:38:05.081007 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.081015    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"
Sep 05 02:38:05.081424 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.081061    2555 scope.go:117] "RemoveContainer" containerID="cdd995203779303de7f9cd9888a85b445f700636de1c8f8e96372a8ca274b03b"
Sep 05 02:38:05.083416 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.083386    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:05.089863 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.089842    2555 scope.go:117] "RemoveContainer" containerID="5aaeadd7568ef9aa59439b573f7955a116459951e5f6d760622fc4be59e3f9e9"
Sep 05 02:38:05.097008 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.096992    2555 scope.go:117] "RemoveContainer" containerID="1ebfb0cab4b521759441ba5e001eb7f8e089996d98fda0d9119bcc80bb7a3768"
Sep 05 02:38:05.137979 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.137951    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"]
Sep 05 02:38:05.142650 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.142619    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-4smct"]
Sep 05 02:38:05.390645 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.390619    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"]
Sep 05 02:38:05.393142 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:38:05.393106    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd37cf1_dc7d_49b6_97e7_5a0d9b3ee667.slice/crio-4d8700e99f3fc2a0ab6cb0916edf41c25d656495cd380d08df97913d7522fa56 WatchSource:0}: Error finding container 4d8700e99f3fc2a0ab6cb0916edf41c25d656495cd380d08df97913d7522fa56: Status 404 returned error can't find the container with id 4d8700e99f3fc2a0ab6cb0916edf41c25d656495cd380d08df97913d7522fa56
Sep 05 02:38:05.544316 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:05.544277    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6466c540-7eef-459e-99bd-f67a1beb1fdd" path="/var/lib/kubelet/pods/6466c540-7eef-459e-99bd-f67a1beb1fdd/volumes"
Sep 05 02:38:06.087087 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:06.087053    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" event={"ID":"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667","Type":"ContainerStarted","Data":"44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05"}
Sep 05 02:38:06.087518 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:06.087115    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" event={"ID":"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667","Type":"ContainerStarted","Data":"4d8700e99f3fc2a0ab6cb0916edf41c25d656495cd380d08df97913d7522fa56"}
Sep 05 02:38:26.177923 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:26.177888    2555 generic.go:356] "Generic (PLEG): container finished" podID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerID="44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05" exitCode=0
Sep 05 02:38:26.178373 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:26.177972    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" event={"ID":"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667","Type":"ContainerDied","Data":"44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05"}
Sep 05 02:38:27.184067 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:27.184027    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" event={"ID":"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667","Type":"ContainerStarted","Data":"e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278"}
Sep 05 02:38:27.184490 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:27.184093    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" event={"ID":"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667","Type":"ContainerStarted","Data":"f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607"}
Sep 05 02:38:27.184490 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:27.184287    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:27.205178 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:27.205130    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" podStartSLOduration=2.691388904 podStartE2EDuration="23.205114206s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.513725302s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:38:27.203581391 +0000 UTC m=+1228.212077548" watchObservedRunningTime="2026-09-05 02:38:27.205114206 +0000 UTC m=+1228.213610344"
Sep 05 02:38:28.188487 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:28.188457    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:38:34.200661 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:38:34.200631    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:39:57.544399 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:39:57.544372    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:40:08.581997 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:08.581947    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 02:40:10.923118 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:10.923078    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"]
Sep 05 02:40:10.923514 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:10.923330    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerName="kserve-container" containerID="cri-o://f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607" gracePeriod=30
Sep 05 02:40:10.923514 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:10.923372    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerName="kube-rbac-proxy" containerID="cri-o://e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278" gracePeriod=30
Sep 05 02:40:11.666669 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:11.666628    2555 generic.go:356] "Generic (PLEG): container finished" podID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerID="e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278" exitCode=2
Sep 05 02:40:11.666840 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:11.666702    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" event={"ID":"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667","Type":"ContainerDied","Data":"e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278"}
Sep 05 02:40:12.359474 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.359449    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:40:12.413367 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.413331    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-rbac-proxy-sar-config\") pod \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") "
Sep 05 02:40:12.413576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.413430    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-proxy-tls\") pod \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") "
Sep 05 02:40:12.413576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.413457    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-cabundle-cert\") pod \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") "
Sep 05 02:40:12.413576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.413481    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8428b\" (UniqueName: \"kubernetes.io/projected/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-api-access-8428b\") pod \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") "
Sep 05 02:40:12.413576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.413503    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kserve-provision-location\") pod \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\" (UID: \"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667\") "
Sep 05 02:40:12.413849 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.413750    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" (UID: "3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:12.413956 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.413818    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" (UID: "3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:12.415602 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.415580    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" (UID: "3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:40:12.415692 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.415642    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-api-access-8428b" (OuterVolumeSpecName: "kube-api-access-8428b") pod "3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" (UID: "3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667"). InnerVolumeSpecName "kube-api-access-8428b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:40:12.417620 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.417595    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" (UID: "3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:40:12.514946 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.514839    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:40:12.514946 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.514886    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8428b\" (UniqueName: \"kubernetes.io/projected/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-api-access-8428b\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:40:12.514946 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.514896    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:40:12.514946 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.514905    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:40:12.514946 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.514915    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:40:12.673460 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.673421    2555 generic.go:356] "Generic (PLEG): container finished" podID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerID="f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607" exitCode=0
Sep 05 02:40:12.673647 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.673486    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"
Sep 05 02:40:12.673647 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.673498    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" event={"ID":"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667","Type":"ContainerDied","Data":"f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607"}
Sep 05 02:40:12.673647 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.673592    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm" event={"ID":"3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667","Type":"ContainerDied","Data":"4d8700e99f3fc2a0ab6cb0916edf41c25d656495cd380d08df97913d7522fa56"}
Sep 05 02:40:12.673647 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.673637    2555 scope.go:117] "RemoveContainer" containerID="e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278"
Sep 05 02:40:12.682020 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.682002    2555 scope.go:117] "RemoveContainer" containerID="f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607"
Sep 05 02:40:12.689303 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.689287    2555 scope.go:117] "RemoveContainer" containerID="44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05"
Sep 05 02:40:12.694956 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.694934    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"]
Sep 05 02:40:12.696875 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.696854    2555 scope.go:117] "RemoveContainer" containerID="e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278"
Sep 05 02:40:12.697177 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:40:12.697156    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278\": container with ID starting with e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278 not found: ID does not exist" containerID="e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278"
Sep 05 02:40:12.697241 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.697189    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278"} err="failed to get container status \"e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278\": rpc error: code = NotFound desc = could not find container \"e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278\": container with ID starting with e8c0637c075950b79a327f355332221924a05b765cb84373bad234cb7f9da278 not found: ID does not exist"
Sep 05 02:40:12.697241 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.697208    2555 scope.go:117] "RemoveContainer" containerID="f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607"
Sep 05 02:40:12.697509 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:40:12.697492    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607\": container with ID starting with f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607 not found: ID does not exist" containerID="f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607"
Sep 05 02:40:12.697559 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.697517    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607"} err="failed to get container status \"f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607\": rpc error: code = NotFound desc = could not find container \"f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607\": container with ID starting with f4d29683dfc8799a65e929b1b48f74fc189d1a434366cde953c03810e4f60607 not found: ID does not exist"
Sep 05 02:40:12.697559 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.697557    2555 scope.go:117] "RemoveContainer" containerID="44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05"
Sep 05 02:40:12.697786 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:40:12.697765    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05\": container with ID starting with 44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05 not found: ID does not exist" containerID="44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05"
Sep 05 02:40:12.697822 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.697792    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05"} err="failed to get container status \"44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05\": rpc error: code = NotFound desc = could not find container \"44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05\": container with ID starting with 44df7846b4da844a4392ade89026652f2cded372574feec33a3f4eb1df7f6e05 not found: ID does not exist"
Sep 05 02:40:12.701326 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:12.701304    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bg7sm"]
Sep 05 02:40:13.543906 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:13.543864    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" path="/var/lib/kubelet/pods/3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667/volumes"
Sep 05 02:40:16.123589 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.123556    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"]
Sep 05 02:40:16.123957 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.123890    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerName="kube-rbac-proxy"
Sep 05 02:40:16.123957 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.123901    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerName="kube-rbac-proxy"
Sep 05 02:40:16.123957 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.123915    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerName="kserve-container"
Sep 05 02:40:16.123957 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.123920    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerName="kserve-container"
Sep 05 02:40:16.123957 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.123928    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerName="storage-initializer"
Sep 05 02:40:16.123957 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.123934    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerName="storage-initializer"
Sep 05 02:40:16.124154 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.123989    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerName="kube-rbac-proxy"
Sep 05 02:40:16.124154 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.123996    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fd37cf1-dc7d-49b6-97e7-5a0d9b3ee667" containerName="kserve-container"
Sep 05 02:40:16.128661 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.128636    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.131566 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.131526    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:40:16.132017 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.131999    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:40:16.132424 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.132405    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:40:16.132527 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.132436    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 05 02:40:16.132527 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.132456    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 05 02:40:16.132527 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.132463    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:40:16.132527 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.132440    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:40:16.140664 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.140628    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"]
Sep 05 02:40:16.250173 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.250111    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.250173 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.250174    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8r69\" (UniqueName: \"kubernetes.io/projected/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-api-access-n8r69\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.250400 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.250225    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.250400 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.250297    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f75e43f-b37b-4e31-ba70-6c11836a66d8-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.250400 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.250332    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.351498 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.351442    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.351498 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.351502    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.351784 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.351524    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8r69\" (UniqueName: \"kubernetes.io/projected/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-api-access-n8r69\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.351784 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.351566    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.351784 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.351628    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f75e43f-b37b-4e31-ba70-6c11836a66d8-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.351784 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:40:16.351720    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 05 02:40:16.351993 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:40:16.351789    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f75e43f-b37b-4e31-ba70-6c11836a66d8-proxy-tls podName:8f75e43f-b37b-4e31-ba70-6c11836a66d8 nodeName:}" failed. No retries permitted until 2026-09-05 02:40:16.851769062 +0000 UTC m=+1337.860265179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8f75e43f-b37b-4e31-ba70-6c11836a66d8-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" (UID: "8f75e43f-b37b-4e31-ba70-6c11836a66d8") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 05 02:40:16.352068 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.352044    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.352275 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.352258    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.352314 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.352258    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.360469 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.360447    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8r69\" (UniqueName: \"kubernetes.io/projected/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-api-access-n8r69\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.857422 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.857382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f75e43f-b37b-4e31-ba70-6c11836a66d8-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:16.859769 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:16.859738    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f75e43f-b37b-4e31-ba70-6c11836a66d8-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:17.039001 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:17.038958    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:17.314253 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:17.314227    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"]
Sep 05 02:40:17.316829 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:40:17.316801    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f75e43f_b37b_4e31_ba70_6c11836a66d8.slice/crio-5d07974906d011ccf4fb359879bb1711b815cb477ab9aca86ef0ae29940ba9d2 WatchSource:0}: Error finding container 5d07974906d011ccf4fb359879bb1711b815cb477ab9aca86ef0ae29940ba9d2: Status 404 returned error can't find the container with id 5d07974906d011ccf4fb359879bb1711b815cb477ab9aca86ef0ae29940ba9d2
Sep 05 02:40:17.697139 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:17.697099    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" event={"ID":"8f75e43f-b37b-4e31-ba70-6c11836a66d8","Type":"ContainerStarted","Data":"00551d4f8e9d65bd994e835c4f4e17fcac3a630c49fc65f214012905736ef6e3"}
Sep 05 02:40:17.697351 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:17.697171    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" event={"ID":"8f75e43f-b37b-4e31-ba70-6c11836a66d8","Type":"ContainerStarted","Data":"5d07974906d011ccf4fb359879bb1711b815cb477ab9aca86ef0ae29940ba9d2"}
Sep 05 02:40:23.725261 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:23.725174    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerID="00551d4f8e9d65bd994e835c4f4e17fcac3a630c49fc65f214012905736ef6e3" exitCode=0
Sep 05 02:40:23.725261 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:23.725247    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" event={"ID":"8f75e43f-b37b-4e31-ba70-6c11836a66d8","Type":"ContainerDied","Data":"00551d4f8e9d65bd994e835c4f4e17fcac3a630c49fc65f214012905736ef6e3"}
Sep 05 02:40:24.732778 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:24.732739    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" event={"ID":"8f75e43f-b37b-4e31-ba70-6c11836a66d8","Type":"ContainerStarted","Data":"7f5f1de0a8a8c93f768336becf228a9eec095fc0f052207eb82b663c4fd138c4"}
Sep 05 02:40:24.733244 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:24.732849    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" event={"ID":"8f75e43f-b37b-4e31-ba70-6c11836a66d8","Type":"ContainerStarted","Data":"5f72e3f5523bc69979ef463fd9e28ad299963eadec3b61e13895b92de6cf8ff5"}
Sep 05 02:40:24.733244 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:24.733114    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:24.733357 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:24.733256    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:40:24.753249 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:24.753193    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" podStartSLOduration=2.668653383 podStartE2EDuration="8.753176474s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.084523091s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:40:24.751453622 +0000 UTC m=+1345.759949761" watchObservedRunningTime="2026-09-05 02:40:24.753176474 +0000 UTC m=+1345.761672612"
Sep 05 02:40:30.743227 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:40:30.743197    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:41:59.546727 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:41:59.546635    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:42:10.583711 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:10.583662    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.35:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 02:42:12.812936 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:12.812902    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"]
Sep 05 02:42:12.813338 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:12.813155    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerName="kserve-container" containerID="cri-o://5f72e3f5523bc69979ef463fd9e28ad299963eadec3b61e13895b92de6cf8ff5" gracePeriod=30
Sep 05 02:42:12.813338 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:12.813193    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerName="kube-rbac-proxy" containerID="cri-o://7f5f1de0a8a8c93f768336becf228a9eec095fc0f052207eb82b663c4fd138c4" gracePeriod=30
Sep 05 02:42:13.222988 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:13.222953    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerID="7f5f1de0a8a8c93f768336becf228a9eec095fc0f052207eb82b663c4fd138c4" exitCode=2
Sep 05 02:42:13.223171 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:13.223027    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" event={"ID":"8f75e43f-b37b-4e31-ba70-6c11836a66d8","Type":"ContainerDied","Data":"7f5f1de0a8a8c93f768336becf228a9eec095fc0f052207eb82b663c4fd138c4"}
Sep 05 02:42:14.229801 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.229766    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerID="5f72e3f5523bc69979ef463fd9e28ad299963eadec3b61e13895b92de6cf8ff5" exitCode=0
Sep 05 02:42:14.229801 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.229809    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" event={"ID":"8f75e43f-b37b-4e31-ba70-6c11836a66d8","Type":"ContainerDied","Data":"5f72e3f5523bc69979ef463fd9e28ad299963eadec3b61e13895b92de6cf8ff5"}
Sep 05 02:42:14.261034 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.261011    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:42:14.367961 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.367876    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f75e43f-b37b-4e31-ba70-6c11836a66d8-proxy-tls\") pod \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") "
Sep 05 02:42:14.367961 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.367944    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-cabundle-cert\") pod \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") "
Sep 05 02:42:14.368153 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.368014    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8r69\" (UniqueName: \"kubernetes.io/projected/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-api-access-n8r69\") pod \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") "
Sep 05 02:42:14.368153 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.368068    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-rbac-proxy-sar-config\") pod \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") "
Sep 05 02:42:14.368153 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.368116    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kserve-provision-location\") pod \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\" (UID: \"8f75e43f-b37b-4e31-ba70-6c11836a66d8\") "
Sep 05 02:42:14.368321 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.368233    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f75e43f-b37b-4e31-ba70-6c11836a66d8" (UID: "8f75e43f-b37b-4e31-ba70-6c11836a66d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:42:14.368484 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.368466    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:42:14.368578 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.368474    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f75e43f-b37b-4e31-ba70-6c11836a66d8" (UID: "8f75e43f-b37b-4e31-ba70-6c11836a66d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:42:14.370043 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.370023    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-api-access-n8r69" (OuterVolumeSpecName: "kube-api-access-n8r69") pod "8f75e43f-b37b-4e31-ba70-6c11836a66d8" (UID: "8f75e43f-b37b-4e31-ba70-6c11836a66d8"). InnerVolumeSpecName "kube-api-access-n8r69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:42:14.370291 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.370274    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f75e43f-b37b-4e31-ba70-6c11836a66d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f75e43f-b37b-4e31-ba70-6c11836a66d8" (UID: "8f75e43f-b37b-4e31-ba70-6c11836a66d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:42:14.373458 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.373431    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f75e43f-b37b-4e31-ba70-6c11836a66d8" (UID: "8f75e43f-b37b-4e31-ba70-6c11836a66d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:42:14.469426 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.469394    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f75e43f-b37b-4e31-ba70-6c11836a66d8-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:42:14.469426 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.469419    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8r69\" (UniqueName: \"kubernetes.io/projected/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-api-access-n8r69\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:42:14.469426 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.469431    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:42:14.469662 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:14.469440    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f75e43f-b37b-4e31-ba70-6c11836a66d8-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:42:15.236124 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:15.236098    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"
Sep 05 02:42:15.236611 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:15.236091    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2" event={"ID":"8f75e43f-b37b-4e31-ba70-6c11836a66d8","Type":"ContainerDied","Data":"5d07974906d011ccf4fb359879bb1711b815cb477ab9aca86ef0ae29940ba9d2"}
Sep 05 02:42:15.236611 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:15.236261    2555 scope.go:117] "RemoveContainer" containerID="7f5f1de0a8a8c93f768336becf228a9eec095fc0f052207eb82b663c4fd138c4"
Sep 05 02:42:15.245098 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:15.245080    2555 scope.go:117] "RemoveContainer" containerID="5f72e3f5523bc69979ef463fd9e28ad299963eadec3b61e13895b92de6cf8ff5"
Sep 05 02:42:15.252667 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:15.252652    2555 scope.go:117] "RemoveContainer" containerID="00551d4f8e9d65bd994e835c4f4e17fcac3a630c49fc65f214012905736ef6e3"
Sep 05 02:42:15.259299 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:15.259268    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"]
Sep 05 02:42:15.262945 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:15.262925    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-dj4g2"]
Sep 05 02:42:15.543942 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:15.543861    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" path="/var/lib/kubelet/pods/8f75e43f-b37b-4e31-ba70-6c11836a66d8/volumes"
Sep 05 02:42:17.988288 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.988254    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"]
Sep 05 02:42:17.988716 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.988596    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerName="kube-rbac-proxy"
Sep 05 02:42:17.988716 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.988609    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerName="kube-rbac-proxy"
Sep 05 02:42:17.988716 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.988620    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerName="storage-initializer"
Sep 05 02:42:17.988716 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.988626    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerName="storage-initializer"
Sep 05 02:42:17.988716 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.988634    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerName="kserve-container"
Sep 05 02:42:17.988716 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.988639    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerName="kserve-container"
Sep 05 02:42:17.988716 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.988687    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerName="kube-rbac-proxy"
Sep 05 02:42:17.988716 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.988695    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f75e43f-b37b-4e31-ba70-6c11836a66d8" containerName="kserve-container"
Sep 05 02:42:17.993130 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.993112    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:17.995701 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.995676    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 05 02:42:17.995701 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.995699    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:42:17.996004 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.995989    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:42:17.996072 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.996000    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:42:17.996072 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.996047    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:42:17.996192 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.996001    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 05 02:42:17.997219 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:17.997198    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:42:18.003581 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.003553    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"]
Sep 05 02:42:18.096431 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.096392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.096645 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.096436    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.096645 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.096467    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.096645 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.096575    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.096776 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.096644    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rr5\" (UniqueName: \"kubernetes.io/projected/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-api-access-76rr5\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.198045 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.198003    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.198215 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.198070    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.198215 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.198126    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rr5\" (UniqueName: \"kubernetes.io/projected/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-api-access-76rr5\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.198215 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:42:18.198168    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 05 02:42:18.198385 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:42:18.198239    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-proxy-tls podName:d62dec9e-00b6-4fcd-ba85-661a7e56dff0 nodeName:}" failed. No retries permitted until 2026-09-05 02:42:18.698216871 +0000 UTC m=+1459.706713001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" (UID: "d62dec9e-00b6-4fcd-ba85-661a7e56dff0") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 05 02:42:18.198385 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.198258    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.198385 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.198298    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.198696 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.198675    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.198837 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.198819    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.198879 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.198839    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.206792 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.206769    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rr5\" (UniqueName: \"kubernetes.io/projected/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-api-access-76rr5\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.702794 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.702757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.705126 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.705108    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:18.904912 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:18.904870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:19.177766 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:19.177665    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"]
Sep 05 02:42:19.180007 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:42:19.179977    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62dec9e_00b6_4fcd_ba85_661a7e56dff0.slice/crio-e05ccc4b1fd7a57770890229d8a98d84cdc771fe2d9434572b0df33369341d82 WatchSource:0}: Error finding container e05ccc4b1fd7a57770890229d8a98d84cdc771fe2d9434572b0df33369341d82: Status 404 returned error can't find the container with id e05ccc4b1fd7a57770890229d8a98d84cdc771fe2d9434572b0df33369341d82
Sep 05 02:42:19.256681 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:19.256642    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" event={"ID":"d62dec9e-00b6-4fcd-ba85-661a7e56dff0","Type":"ContainerStarted","Data":"9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518"}
Sep 05 02:42:19.256822 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:19.256713    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" event={"ID":"d62dec9e-00b6-4fcd-ba85-661a7e56dff0","Type":"ContainerStarted","Data":"e05ccc4b1fd7a57770890229d8a98d84cdc771fe2d9434572b0df33369341d82"}
Sep 05 02:42:49.387295 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:49.387203    2555 generic.go:356] "Generic (PLEG): container finished" podID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerID="9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518" exitCode=0
Sep 05 02:42:49.387295 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:49.387254    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" event={"ID":"d62dec9e-00b6-4fcd-ba85-661a7e56dff0","Type":"ContainerDied","Data":"9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518"}
Sep 05 02:42:50.394151 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:50.394115    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" event={"ID":"d62dec9e-00b6-4fcd-ba85-661a7e56dff0","Type":"ContainerStarted","Data":"1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b"}
Sep 05 02:42:50.394554 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:50.394177    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" event={"ID":"d62dec9e-00b6-4fcd-ba85-661a7e56dff0","Type":"ContainerStarted","Data":"53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf"}
Sep 05 02:42:50.394554 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:50.394352    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:50.419021 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:50.418967    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" podStartSLOduration=3.520024961 podStartE2EDuration="33.418950813s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.898925852s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:42:50.417276646 +0000 UTC m=+1491.425772822" watchObservedRunningTime="2026-09-05 02:42:50.418950813 +0000 UTC m=+1491.427446951"
Sep 05 02:42:51.398897 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:51.398851    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:57.408188 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:57.408158    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:42:59.545500 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:59.545468    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:42:59.546920 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:42:59.546890    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:44:27.544284 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:27.544255    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:44:34.396040 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:34.396009    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"]
Sep 05 02:44:34.396495 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:34.396237    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" podUID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerName="kserve-container" containerID="cri-o://53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf" gracePeriod=30
Sep 05 02:44:34.396495 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:34.396327    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" podUID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerName="kube-rbac-proxy" containerID="cri-o://1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b" gracePeriod=30
Sep 05 02:44:34.865752 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:34.865666    2555 generic.go:356] "Generic (PLEG): container finished" podID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerID="1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b" exitCode=2
Sep 05 02:44:34.865898 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:34.865746    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" event={"ID":"d62dec9e-00b6-4fcd-ba85-661a7e56dff0","Type":"ContainerDied","Data":"1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b"}
Sep 05 02:44:35.744034 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.744011    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:44:35.849516 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.849423    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-cabundle-cert\") pod \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") "
Sep 05 02:44:35.849516 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.849469    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kserve-provision-location\") pod \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") "
Sep 05 02:44:35.849516 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.849508    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-proxy-tls\") pod \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") "
Sep 05 02:44:35.849817 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.849573    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-rbac-proxy-sar-config\") pod \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") "
Sep 05 02:44:35.849817 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.849598    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76rr5\" (UniqueName: \"kubernetes.io/projected/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-api-access-76rr5\") pod \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\" (UID: \"d62dec9e-00b6-4fcd-ba85-661a7e56dff0\") "
Sep 05 02:44:35.849938 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.849907    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d62dec9e-00b6-4fcd-ba85-661a7e56dff0" (UID: "d62dec9e-00b6-4fcd-ba85-661a7e56dff0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:44:35.850000 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.849936    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d62dec9e-00b6-4fcd-ba85-661a7e56dff0" (UID: "d62dec9e-00b6-4fcd-ba85-661a7e56dff0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:44:35.851774 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.851747    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d62dec9e-00b6-4fcd-ba85-661a7e56dff0" (UID: "d62dec9e-00b6-4fcd-ba85-661a7e56dff0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:44:35.851869 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.851840    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-api-access-76rr5" (OuterVolumeSpecName: "kube-api-access-76rr5") pod "d62dec9e-00b6-4fcd-ba85-661a7e56dff0" (UID: "d62dec9e-00b6-4fcd-ba85-661a7e56dff0"). InnerVolumeSpecName "kube-api-access-76rr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:44:35.854064 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.854041    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d62dec9e-00b6-4fcd-ba85-661a7e56dff0" (UID: "d62dec9e-00b6-4fcd-ba85-661a7e56dff0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:44:35.871661 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.871631    2555 generic.go:356] "Generic (PLEG): container finished" podID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerID="53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf" exitCode=0
Sep 05 02:44:35.871816 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.871683    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"
Sep 05 02:44:35.871816 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.871716    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" event={"ID":"d62dec9e-00b6-4fcd-ba85-661a7e56dff0","Type":"ContainerDied","Data":"53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf"}
Sep 05 02:44:35.871816 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.871794    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4" event={"ID":"d62dec9e-00b6-4fcd-ba85-661a7e56dff0","Type":"ContainerDied","Data":"e05ccc4b1fd7a57770890229d8a98d84cdc771fe2d9434572b0df33369341d82"}
Sep 05 02:44:35.871983 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.871836    2555 scope.go:117] "RemoveContainer" containerID="1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b"
Sep 05 02:44:35.880868 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.880850    2555 scope.go:117] "RemoveContainer" containerID="53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf"
Sep 05 02:44:35.888555 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.888517    2555 scope.go:117] "RemoveContainer" containerID="9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518"
Sep 05 02:44:35.895056 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.895033    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"]
Sep 05 02:44:35.896455 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.896436    2555 scope.go:117] "RemoveContainer" containerID="1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b"
Sep 05 02:44:35.896751 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:44:35.896730    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b\": container with ID starting with 1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b not found: ID does not exist" containerID="1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b"
Sep 05 02:44:35.896852 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.896762    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b"} err="failed to get container status \"1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b\": rpc error: code = NotFound desc = could not find container \"1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b\": container with ID starting with 1c8ed09ab2aad5c1473b701d34b8adb0acda654cd1c944a6e89ace2d6ab9bb8b not found: ID does not exist"
Sep 05 02:44:35.896920 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.896853    2555 scope.go:117] "RemoveContainer" containerID="53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf"
Sep 05 02:44:35.897698 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:44:35.897674    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf\": container with ID starting with 53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf not found: ID does not exist" containerID="53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf"
Sep 05 02:44:35.897784 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.897702    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf"} err="failed to get container status \"53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf\": rpc error: code = NotFound desc = could not find container \"53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf\": container with ID starting with 53f1b43de71797a20f90643b8337216b6ee8bce10830564f8f100b0ce0d0c2cf not found: ID does not exist"
Sep 05 02:44:35.897784 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.897723    2555 scope.go:117] "RemoveContainer" containerID="9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518"
Sep 05 02:44:35.898006 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:44:35.897982    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518\": container with ID starting with 9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518 not found: ID does not exist" containerID="9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518"
Sep 05 02:44:35.898049 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.898010    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518"} err="failed to get container status \"9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518\": rpc error: code = NotFound desc = could not find container \"9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518\": container with ID starting with 9d35c777710a5de1bbccbbd1a6b3ebcd1df8fd89c27195c8e15609118a04b518 not found: ID does not exist"
Sep 05 02:44:35.899006 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.898988    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fk4p4"]
Sep 05 02:44:35.950595 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.950559    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:44:35.950595 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.950589    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:44:35.950595 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.950599    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:44:35.950807 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.950613    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:44:35.950807 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:35.950621    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76rr5\" (UniqueName: \"kubernetes.io/projected/d62dec9e-00b6-4fcd-ba85-661a7e56dff0-kube-api-access-76rr5\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:44:37.544064 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:37.544025    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" path="/var/lib/kubelet/pods/d62dec9e-00b6-4fcd-ba85-661a7e56dff0/volumes"
Sep 05 02:44:39.477888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.477848    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"]
Sep 05 02:44:39.478256 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.478201    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerName="kserve-container"
Sep 05 02:44:39.478256 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.478213    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerName="kserve-container"
Sep 05 02:44:39.478256 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.478232    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerName="storage-initializer"
Sep 05 02:44:39.478256 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.478237    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerName="storage-initializer"
Sep 05 02:44:39.478256 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.478247    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerName="kube-rbac-proxy"
Sep 05 02:44:39.478256 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.478252    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerName="kube-rbac-proxy"
Sep 05 02:44:39.478449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.478311    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerName="kube-rbac-proxy"
Sep 05 02:44:39.478449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.478319    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62dec9e-00b6-4fcd-ba85-661a7e56dff0" containerName="kserve-container"
Sep 05 02:44:39.483095 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.483075    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.485494 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.485465    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 05 02:44:39.485670 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.485478    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 05 02:44:39.485670 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.485507    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:44:39.485670 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.485480    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:44:39.485670 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.485637    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:44:39.485670 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.485638    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:44:39.485881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.485677    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:44:39.491738 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.491716    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"]
Sep 05 02:44:39.581205 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.581174    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2504ff86-a66c-4e57-bccf-52e2660b882a-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.581387 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.581238    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.581387 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.581286    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmbm\" (UniqueName: \"kubernetes.io/projected/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-api-access-xrmbm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.581387 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.581305    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2504ff86-a66c-4e57-bccf-52e2660b882a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.581387 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.581322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.682572 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.682516    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.682738 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.682603    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmbm\" (UniqueName: \"kubernetes.io/projected/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-api-access-xrmbm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.682778 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.682750    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2504ff86-a66c-4e57-bccf-52e2660b882a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.682814 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.682783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.682848 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.682829    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2504ff86-a66c-4e57-bccf-52e2660b882a-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.682997 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:44:39.682980    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 05 02:44:39.683071 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:44:39.683059    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2504ff86-a66c-4e57-bccf-52e2660b882a-proxy-tls podName:2504ff86-a66c-4e57-bccf-52e2660b882a nodeName:}" failed. No retries permitted until 2026-09-05 02:44:40.183039276 +0000 UTC m=+1601.191535400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2504ff86-a66c-4e57-bccf-52e2660b882a-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" (UID: "2504ff86-a66c-4e57-bccf-52e2660b882a") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 05 02:44:39.683155 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.683136    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2504ff86-a66c-4e57-bccf-52e2660b882a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.683242 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.683222    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.683328 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.683311    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:39.693812 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:39.693797    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmbm\" (UniqueName: \"kubernetes.io/projected/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-api-access-xrmbm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:40.187797 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:40.187756    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2504ff86-a66c-4e57-bccf-52e2660b882a-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:40.190148 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:40.190120    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2504ff86-a66c-4e57-bccf-52e2660b882a-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:40.394713 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:40.394665    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:40.665793 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:40.665767    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"]
Sep 05 02:44:40.667975 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:44:40.667943    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2504ff86_a66c_4e57_bccf_52e2660b882a.slice/crio-d0bfb209417b54fbc1cb586ddeeb75f83f60ea728faffba519df473683882884 WatchSource:0}: Error finding container d0bfb209417b54fbc1cb586ddeeb75f83f60ea728faffba519df473683882884: Status 404 returned error can't find the container with id d0bfb209417b54fbc1cb586ddeeb75f83f60ea728faffba519df473683882884
Sep 05 02:44:40.897239 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:40.897197    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" event={"ID":"2504ff86-a66c-4e57-bccf-52e2660b882a","Type":"ContainerStarted","Data":"3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4"}
Sep 05 02:44:40.897434 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:40.897269    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" event={"ID":"2504ff86-a66c-4e57-bccf-52e2660b882a","Type":"ContainerStarted","Data":"d0bfb209417b54fbc1cb586ddeeb75f83f60ea728faffba519df473683882884"}
Sep 05 02:44:49.936066 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:49.936030    2555 generic.go:356] "Generic (PLEG): container finished" podID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerID="3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4" exitCode=0
Sep 05 02:44:49.936066 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:49.936071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" event={"ID":"2504ff86-a66c-4e57-bccf-52e2660b882a","Type":"ContainerDied","Data":"3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4"}
Sep 05 02:44:50.943629 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:50.943593    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" event={"ID":"2504ff86-a66c-4e57-bccf-52e2660b882a","Type":"ContainerStarted","Data":"046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054"}
Sep 05 02:44:50.944107 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:50.943703    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" event={"ID":"2504ff86-a66c-4e57-bccf-52e2660b882a","Type":"ContainerStarted","Data":"a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c"}
Sep 05 02:44:50.944107 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:50.943938    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:50.966802 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:50.966738    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" podStartSLOduration=3.147022963 podStartE2EDuration="11.966715315s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.819692352s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:44:50.964761785 +0000 UTC m=+1611.973257947" watchObservedRunningTime="2026-09-05 02:44:50.966715315 +0000 UTC m=+1611.975211454"
Sep 05 02:44:51.948347 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:51.948267    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:51.949963 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:51.949932    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 05 02:44:52.953467 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:52.953430    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 05 02:44:57.958455 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:57.958423    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:44:57.960900 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:44:57.959040    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 05 02:45:07.960364 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:07.960332    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:45:21.868750 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:21.868717    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"]
Sep 05 02:45:21.869174 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:21.868947    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kserve-container" containerID="cri-o://a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c" gracePeriod=30
Sep 05 02:45:21.869174 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:21.869018    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kube-rbac-proxy" containerID="cri-o://046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054" gracePeriod=30
Sep 05 02:45:22.093641 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:22.093606    2555 generic.go:356] "Generic (PLEG): container finished" podID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerID="046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054" exitCode=2
Sep 05 02:45:22.093823 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:22.093688    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" event={"ID":"2504ff86-a66c-4e57-bccf-52e2660b882a","Type":"ContainerDied","Data":"046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054"}
Sep 05 02:45:22.954592 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:22.954529    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 05 02:45:23.606563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.606523    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:45:23.668238 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.668190    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2504ff86-a66c-4e57-bccf-52e2660b882a-kserve-provision-location\") pod \"2504ff86-a66c-4e57-bccf-52e2660b882a\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") "
Sep 05 02:45:23.668420 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.668283    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2504ff86-a66c-4e57-bccf-52e2660b882a-proxy-tls\") pod \"2504ff86-a66c-4e57-bccf-52e2660b882a\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") "
Sep 05 02:45:23.668420 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.668334    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-rbac-proxy-sar-config\") pod \"2504ff86-a66c-4e57-bccf-52e2660b882a\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") "
Sep 05 02:45:23.668740 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.668712    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2504ff86-a66c-4e57-bccf-52e2660b882a" (UID: "2504ff86-a66c-4e57-bccf-52e2660b882a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:45:23.669112 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.669089    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2504ff86-a66c-4e57-bccf-52e2660b882a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2504ff86-a66c-4e57-bccf-52e2660b882a" (UID: "2504ff86-a66c-4e57-bccf-52e2660b882a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:23.670280 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.670259    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2504ff86-a66c-4e57-bccf-52e2660b882a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2504ff86-a66c-4e57-bccf-52e2660b882a" (UID: "2504ff86-a66c-4e57-bccf-52e2660b882a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:45:23.769387 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.769294    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-cabundle-cert\") pod \"2504ff86-a66c-4e57-bccf-52e2660b882a\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") "
Sep 05 02:45:23.769387 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.769338    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrmbm\" (UniqueName: \"kubernetes.io/projected/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-api-access-xrmbm\") pod \"2504ff86-a66c-4e57-bccf-52e2660b882a\" (UID: \"2504ff86-a66c-4e57-bccf-52e2660b882a\") "
Sep 05 02:45:23.769630 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.769553    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2504ff86-a66c-4e57-bccf-52e2660b882a-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:45:23.769630 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.769572    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2504ff86-a66c-4e57-bccf-52e2660b882a-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:45:23.769630 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.769584    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:45:23.769630 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.769609    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2504ff86-a66c-4e57-bccf-52e2660b882a" (UID: "2504ff86-a66c-4e57-bccf-52e2660b882a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:45:23.771290 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.771270    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-api-access-xrmbm" (OuterVolumeSpecName: "kube-api-access-xrmbm") pod "2504ff86-a66c-4e57-bccf-52e2660b882a" (UID: "2504ff86-a66c-4e57-bccf-52e2660b882a"). InnerVolumeSpecName "kube-api-access-xrmbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:45:23.870392 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.870350    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2504ff86-a66c-4e57-bccf-52e2660b882a-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:45:23.870392 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:23.870383    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrmbm\" (UniqueName: \"kubernetes.io/projected/2504ff86-a66c-4e57-bccf-52e2660b882a-kube-api-access-xrmbm\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:45:24.105229 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.105140    2555 generic.go:356] "Generic (PLEG): container finished" podID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerID="a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c" exitCode=0
Sep 05 02:45:24.105229 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.105195    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"
Sep 05 02:45:24.105229 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.105216    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" event={"ID":"2504ff86-a66c-4e57-bccf-52e2660b882a","Type":"ContainerDied","Data":"a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c"}
Sep 05 02:45:24.105736 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.105272    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql" event={"ID":"2504ff86-a66c-4e57-bccf-52e2660b882a","Type":"ContainerDied","Data":"d0bfb209417b54fbc1cb586ddeeb75f83f60ea728faffba519df473683882884"}
Sep 05 02:45:24.105736 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.105299    2555 scope.go:117] "RemoveContainer" containerID="046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054"
Sep 05 02:45:24.113552 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.113520    2555 scope.go:117] "RemoveContainer" containerID="a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c"
Sep 05 02:45:24.121077 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.121056    2555 scope.go:117] "RemoveContainer" containerID="3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4"
Sep 05 02:45:24.126373 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.126352    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"]
Sep 05 02:45:24.128798 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.128774    2555 scope.go:117] "RemoveContainer" containerID="046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054"
Sep 05 02:45:24.129120 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:45:24.129099    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054\": container with ID starting with 046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054 not found: ID does not exist" containerID="046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054"
Sep 05 02:45:24.129213 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.129129    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054"} err="failed to get container status \"046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054\": rpc error: code = NotFound desc = could not find container \"046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054\": container with ID starting with 046e0a64a8718ad5d4a1eaa72bede7e53c109e0712e68b9288c7afd0523a4054 not found: ID does not exist"
Sep 05 02:45:24.129213 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.129149    2555 scope.go:117] "RemoveContainer" containerID="a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c"
Sep 05 02:45:24.129435 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:45:24.129417    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c\": container with ID starting with a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c not found: ID does not exist" containerID="a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c"
Sep 05 02:45:24.129498 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.129444    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c"} err="failed to get container status \"a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c\": rpc error: code = NotFound desc = could not find container \"a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c\": container with ID starting with a6eb6c1f806c10f1c8ad33901851645dfd5bc437f10935f791486200ad73350c not found: ID does not exist"
Sep 05 02:45:24.129498 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.129466    2555 scope.go:117] "RemoveContainer" containerID="3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4"
Sep 05 02:45:24.129783 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:45:24.129767    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4\": container with ID starting with 3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4 not found: ID does not exist" containerID="3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4"
Sep 05 02:45:24.129820 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.129786    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4"} err="failed to get container status \"3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4\": rpc error: code = NotFound desc = could not find container \"3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4\": container with ID starting with 3fc8e1f4b97c09663405b3d8412b46df812c90e9fed498d4abb37711cd3a74b4 not found: ID does not exist"
Sep 05 02:45:24.130077 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:24.130060    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-v4mql"]
Sep 05 02:45:25.543803 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:25.543764    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" path="/var/lib/kubelet/pods/2504ff86-a66c-4e57-bccf-52e2660b882a/volumes"
Sep 05 02:45:27.045784 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.045745    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"]
Sep 05 02:45:27.046194 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.046088    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kube-rbac-proxy"
Sep 05 02:45:27.046194 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.046102    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kube-rbac-proxy"
Sep 05 02:45:27.046194 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.046120    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="storage-initializer"
Sep 05 02:45:27.046194 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.046127    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="storage-initializer"
Sep 05 02:45:27.046194 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.046139    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kserve-container"
Sep 05 02:45:27.046194 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.046144    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kserve-container"
Sep 05 02:45:27.046194 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.046195    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kube-rbac-proxy"
Sep 05 02:45:27.046448 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.046206    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2504ff86-a66c-4e57-bccf-52e2660b882a" containerName="kserve-container"
Sep 05 02:45:27.050987 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.050968    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.053473 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.053445    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:45:27.053634 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.053508    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:45:27.053634 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.053545    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:45:27.053634 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.053566    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 05 02:45:27.053634 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.053551    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:45:27.054186 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.054166    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:45:27.054186 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.054176    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 05 02:45:27.061088 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.061064    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"]
Sep 05 02:45:27.094739 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.094703    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.094944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.094743    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx6kz\" (UniqueName: \"kubernetes.io/projected/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-api-access-kx6kz\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.094944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.094779    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0710d9-d1ad-4780-82ba-c69ecbef5937-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.094944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.094813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.094944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.094834    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.195281 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.195248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.195281 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.195283    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx6kz\" (UniqueName: \"kubernetes.io/projected/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-api-access-kx6kz\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.195511 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.195303    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0710d9-d1ad-4780-82ba-c69ecbef5937-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.195511 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.195325    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.195511 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:45:27.195409    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 05 02:45:27.195511 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.195445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.195511 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:45:27.195476    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0710d9-d1ad-4780-82ba-c69ecbef5937-proxy-tls podName:0f0710d9-d1ad-4780-82ba-c69ecbef5937 nodeName:}" failed. No retries permitted until 2026-09-05 02:45:27.695455946 +0000 UTC m=+1648.703952077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0f0710d9-d1ad-4780-82ba-c69ecbef5937-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" (UID: "0f0710d9-d1ad-4780-82ba-c69ecbef5937") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 05 02:45:27.195887 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.195861    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.195987 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.195967    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.196153 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.196133    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.204206 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.204183    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx6kz\" (UniqueName: \"kubernetes.io/projected/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-api-access-kx6kz\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.698326 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.698286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0710d9-d1ad-4780-82ba-c69ecbef5937-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.700672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.700647    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0710d9-d1ad-4780-82ba-c69ecbef5937-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:27.961977 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:27.961885    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:28.233756 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:28.233727    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"]
Sep 05 02:45:28.236240 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:45:28.236207    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0710d9_d1ad_4780_82ba_c69ecbef5937.slice/crio-16aeae5dad7bb4931d70f5077d296d01d4b2b7ee40d06ea2b68c7f20bc8f2750 WatchSource:0}: Error finding container 16aeae5dad7bb4931d70f5077d296d01d4b2b7ee40d06ea2b68c7f20bc8f2750: Status 404 returned error can't find the container with id 16aeae5dad7bb4931d70f5077d296d01d4b2b7ee40d06ea2b68c7f20bc8f2750
Sep 05 02:45:29.136564 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:29.136501    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" event={"ID":"0f0710d9-d1ad-4780-82ba-c69ecbef5937","Type":"ContainerStarted","Data":"84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324"}
Sep 05 02:45:29.136797 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:29.136604    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" event={"ID":"0f0710d9-d1ad-4780-82ba-c69ecbef5937","Type":"ContainerStarted","Data":"16aeae5dad7bb4931d70f5077d296d01d4b2b7ee40d06ea2b68c7f20bc8f2750"}
Sep 05 02:45:33.156175 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:33.156133    2555 generic.go:356] "Generic (PLEG): container finished" podID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerID="84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324" exitCode=0
Sep 05 02:45:33.156598 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:33.156211    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" event={"ID":"0f0710d9-d1ad-4780-82ba-c69ecbef5937","Type":"ContainerDied","Data":"84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324"}
Sep 05 02:45:34.163122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:34.163084    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" event={"ID":"0f0710d9-d1ad-4780-82ba-c69ecbef5937","Type":"ContainerStarted","Data":"7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2"}
Sep 05 02:45:34.163619 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:34.163152    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" event={"ID":"0f0710d9-d1ad-4780-82ba-c69ecbef5937","Type":"ContainerStarted","Data":"e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a"}
Sep 05 02:45:34.163619 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:34.163490    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:34.163768 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:34.163633    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:34.165335 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:34.165307    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 05 02:45:34.193554 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:34.193483    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" podStartSLOduration=2.758521043 podStartE2EDuration="7.193465523s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.43494448s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:45:34.191087882 +0000 UTC m=+1655.199584041" watchObservedRunningTime="2026-09-05 02:45:34.193465523 +0000 UTC m=+1655.201961662"
Sep 05 02:45:35.168161 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:35.168122    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 05 02:45:40.173114 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:40.173080    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:45:40.173721 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:40.173692    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 05 02:45:50.174731 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:45:50.174703    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:46:02.293394 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:02.293359    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"]
Sep 05 02:46:02.293817 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:02.293622    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="kserve-container" containerID="cri-o://e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a" gracePeriod=30
Sep 05 02:46:02.293817 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:02.293652    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="kube-rbac-proxy" containerID="cri-o://7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2" gracePeriod=30
Sep 05 02:46:03.299123 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.299088    2555 generic.go:356] "Generic (PLEG): container finished" podID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerID="7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2" exitCode=2
Sep 05 02:46:03.299607 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.299160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" event={"ID":"0f0710d9-d1ad-4780-82ba-c69ecbef5937","Type":"ContainerDied","Data":"7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2"}
Sep 05 02:46:03.740598 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.740575    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:46:03.803117 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.803065    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-cabundle-cert\") pod \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") "
Sep 05 02:46:03.803324 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.803308    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kserve-provision-location\") pod \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") "
Sep 05 02:46:03.803398 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.803348    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-rbac-proxy-sar-config\") pod \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") "
Sep 05 02:46:03.803459 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.803407    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0f0710d9-d1ad-4780-82ba-c69ecbef5937" (UID: "0f0710d9-d1ad-4780-82ba-c69ecbef5937"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:46:03.803459 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.803427    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx6kz\" (UniqueName: \"kubernetes.io/projected/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-api-access-kx6kz\") pod \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") "
Sep 05 02:46:03.803590 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.803459    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0710d9-d1ad-4780-82ba-c69ecbef5937-proxy-tls\") pod \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\" (UID: \"0f0710d9-d1ad-4780-82ba-c69ecbef5937\") "
Sep 05 02:46:03.803725 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.803708    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:46:03.803783 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.803761    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0f0710d9-d1ad-4780-82ba-c69ecbef5937" (UID: "0f0710d9-d1ad-4780-82ba-c69ecbef5937"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:46:03.804373 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.804348    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f0710d9-d1ad-4780-82ba-c69ecbef5937" (UID: "0f0710d9-d1ad-4780-82ba-c69ecbef5937"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:46:03.805650 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.805623    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0710d9-d1ad-4780-82ba-c69ecbef5937-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f0710d9-d1ad-4780-82ba-c69ecbef5937" (UID: "0f0710d9-d1ad-4780-82ba-c69ecbef5937"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:46:03.805758 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.805736    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-api-access-kx6kz" (OuterVolumeSpecName: "kube-api-access-kx6kz") pod "0f0710d9-d1ad-4780-82ba-c69ecbef5937" (UID: "0f0710d9-d1ad-4780-82ba-c69ecbef5937"). InnerVolumeSpecName "kube-api-access-kx6kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:46:03.904379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.904345    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx6kz\" (UniqueName: \"kubernetes.io/projected/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-api-access-kx6kz\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:46:03.904379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.904378    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0710d9-d1ad-4780-82ba-c69ecbef5937-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:46:03.904575 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.904388    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:46:03.904575 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:03.904397    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0710d9-d1ad-4780-82ba-c69ecbef5937-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:46:04.305500 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.305405    2555 generic.go:356] "Generic (PLEG): container finished" podID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerID="e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a" exitCode=0
Sep 05 02:46:04.305500 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.305484    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"
Sep 05 02:46:04.305500 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.305487    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" event={"ID":"0f0710d9-d1ad-4780-82ba-c69ecbef5937","Type":"ContainerDied","Data":"e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a"}
Sep 05 02:46:04.306117 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.305599    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm" event={"ID":"0f0710d9-d1ad-4780-82ba-c69ecbef5937","Type":"ContainerDied","Data":"16aeae5dad7bb4931d70f5077d296d01d4b2b7ee40d06ea2b68c7f20bc8f2750"}
Sep 05 02:46:04.306117 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.305629    2555 scope.go:117] "RemoveContainer" containerID="7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2"
Sep 05 02:46:04.314694 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.314674    2555 scope.go:117] "RemoveContainer" containerID="e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a"
Sep 05 02:46:04.321880 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.321863    2555 scope.go:117] "RemoveContainer" containerID="84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324"
Sep 05 02:46:04.328113 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.328092    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"]
Sep 05 02:46:04.329228 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.329211    2555 scope.go:117] "RemoveContainer" containerID="7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2"
Sep 05 02:46:04.329484 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:46:04.329461    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2\": container with ID starting with 7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2 not found: ID does not exist" containerID="7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2"
Sep 05 02:46:04.329529 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.329498    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2"} err="failed to get container status \"7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2\": rpc error: code = NotFound desc = could not find container \"7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2\": container with ID starting with 7622b17d287209c7e26d8547194e46efe11ff0e72dc0eb8ce32fb29ca1c675f2 not found: ID does not exist"
Sep 05 02:46:04.329529 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.329521    2555 scope.go:117] "RemoveContainer" containerID="e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a"
Sep 05 02:46:04.329766 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:46:04.329735    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a\": container with ID starting with e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a not found: ID does not exist" containerID="e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a"
Sep 05 02:46:04.329848 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.329764    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a"} err="failed to get container status \"e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a\": rpc error: code = NotFound desc = could not find container \"e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a\": container with ID starting with e3ee77f7c9ec7552ce133d83cfc0620993b6001194fe1b1e50d58c90275d7e9a not found: ID does not exist"
Sep 05 02:46:04.329848 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.329809    2555 scope.go:117] "RemoveContainer" containerID="84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324"
Sep 05 02:46:04.330190 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:46:04.330166    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324\": container with ID starting with 84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324 not found: ID does not exist" containerID="84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324"
Sep 05 02:46:04.330356 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.330196    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324"} err="failed to get container status \"84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324\": rpc error: code = NotFound desc = could not find container \"84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324\": container with ID starting with 84bbfba59a97dedbd46dcd852f22184ed9da628f7add0467c07261e3b6022324 not found: ID does not exist"
Sep 05 02:46:04.332069 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:04.332049    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-k7dmm"]
Sep 05 02:46:05.543832 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:05.543802    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" path="/var/lib/kubelet/pods/0f0710d9-d1ad-4780-82ba-c69ecbef5937/volumes"
Sep 05 02:46:07.482496 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.482460    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"]
Sep 05 02:46:07.482939 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.482922    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="kserve-container"
Sep 05 02:46:07.482991 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.482942    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="kserve-container"
Sep 05 02:46:07.482991 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.482953    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="kube-rbac-proxy"
Sep 05 02:46:07.482991 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.482959    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="kube-rbac-proxy"
Sep 05 02:46:07.482991 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.482976    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="storage-initializer"
Sep 05 02:46:07.482991 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.482983    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="storage-initializer"
Sep 05 02:46:07.483157 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.483036    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="kserve-container"
Sep 05 02:46:07.483157 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.483045    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f0710d9-d1ad-4780-82ba-c69ecbef5937" containerName="kube-rbac-proxy"
Sep 05 02:46:07.487704 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.487687    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.490392 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.490363    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 05 02:46:07.490517 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.490367    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:46:07.490517 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.490407    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 05 02:46:07.490690 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.490677    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:46:07.490753 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.490738    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:46:07.491462 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.491446    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:46:07.491569 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.491496    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:46:07.496045 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.496026    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"]
Sep 05 02:46:07.522377 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.522344    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"]
Sep 05 02:46:07.525702 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.525685    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.530777 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.530755    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 05 02:46:07.536763 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.536740    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"]
Sep 05 02:46:07.640702 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.640669    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.640702 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.640710    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.640921 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.640741    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.640921 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.640831    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kk8\" (UniqueName: \"kubernetes.io/projected/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-api-access-r4kk8\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.640921 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.640855    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.640921 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.640876    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.641055 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.640934    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.641055 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.640961    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.641055 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.641007    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.641055 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.641028    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9vmf\" (UniqueName: \"kubernetes.io/projected/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-api-access-h9vmf\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.742119 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742034    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.742119 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742074    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.742119 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742113    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.742421 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742154    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kk8\" (UniqueName: \"kubernetes.io/projected/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-api-access-r4kk8\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.742421 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742176    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.742421 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742205    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.742421 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742251    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.742421 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.742421 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742320    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.742421 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742342    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9vmf\" (UniqueName: \"kubernetes.io/projected/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-api-access-h9vmf\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.742800 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742558    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.742800 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.742577    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.743068 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.743041    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.743489 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.743461    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.743845 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.743822    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.744091 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.744067    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.745087 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.745067    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.745398 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.745381    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.751768 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.751745    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kk8\" (UniqueName: \"kubernetes.io/projected/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-api-access-r4kk8\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.751872 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.751791    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9vmf\" (UniqueName: \"kubernetes.io/projected/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-api-access-h9vmf\") pod \"isvc-canary-create-predictor-54dc6cb966-r68mg\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:07.798501 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.798474    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:07.835853 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:07.835784    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:08.072190 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:08.072119    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"]
Sep 05 02:46:08.074379 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:46:08.074355    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cb1fb9_da74_4e3c_ba2e_73cd96c5dcbe.slice/crio-ff83ab09dd5783d73c1baf4f259f027a5cecad648bbd622273936396abed2b11 WatchSource:0}: Error finding container ff83ab09dd5783d73c1baf4f259f027a5cecad648bbd622273936396abed2b11: Status 404 returned error can't find the container with id ff83ab09dd5783d73c1baf4f259f027a5cecad648bbd622273936396abed2b11
Sep 05 02:46:08.116489 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:08.116465    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"]
Sep 05 02:46:08.117962 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:46:08.117939    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccc2a4c_6982_41ac_a8f8_171fde8a3b40.slice/crio-b0a43d88144bfbcf3c3794bb7f79ae026651ced957a2059b5ce2b4edbf0529a9 WatchSource:0}: Error finding container b0a43d88144bfbcf3c3794bb7f79ae026651ced957a2059b5ce2b4edbf0529a9: Status 404 returned error can't find the container with id b0a43d88144bfbcf3c3794bb7f79ae026651ced957a2059b5ce2b4edbf0529a9
Sep 05 02:46:08.326101 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:08.326006    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" event={"ID":"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe","Type":"ContainerStarted","Data":"dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc"}
Sep 05 02:46:08.326101 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:08.326089    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" event={"ID":"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe","Type":"ContainerStarted","Data":"ff83ab09dd5783d73c1baf4f259f027a5cecad648bbd622273936396abed2b11"}
Sep 05 02:46:08.327963 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:08.327934    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" event={"ID":"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40","Type":"ContainerStarted","Data":"be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed"}
Sep 05 02:46:08.328091 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:08.327997    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" event={"ID":"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40","Type":"ContainerStarted","Data":"b0a43d88144bfbcf3c3794bb7f79ae026651ced957a2059b5ce2b4edbf0529a9"}
Sep 05 02:46:12.347614 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:12.347504    2555 generic.go:356] "Generic (PLEG): container finished" podID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerID="be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed" exitCode=0
Sep 05 02:46:12.347614 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:12.347571    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" event={"ID":"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40","Type":"ContainerDied","Data":"be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed"}
Sep 05 02:46:12.348872 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:12.348853    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:46:12.349440 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:12.349380    2555 generic.go:356] "Generic (PLEG): container finished" podID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerID="dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc" exitCode=0
Sep 05 02:46:12.349440 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:12.349431    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" event={"ID":"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe","Type":"ContainerDied","Data":"dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc"}
Sep 05 02:46:28.475934 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:28.475895    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" event={"ID":"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe","Type":"ContainerStarted","Data":"f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2"}
Sep 05 02:46:28.476397 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:28.475979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" event={"ID":"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe","Type":"ContainerStarted","Data":"03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a"}
Sep 05 02:46:28.476397 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:28.476191    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:28.478094 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:28.478066    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" event={"ID":"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40","Type":"ContainerStarted","Data":"b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda"}
Sep 05 02:46:28.478202 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:28.478121    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" event={"ID":"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40","Type":"ContainerStarted","Data":"cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b"}
Sep 05 02:46:28.478310 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:28.478296    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:28.495328 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:28.495277    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podStartSLOduration=1.6926548609999998 podStartE2EDuration="21.495263519s" totalImagesPullingTime="15.954006666s" totalInitContainerRuntime="3.848601992s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:46:28.493721141 +0000 UTC m=+1709.502217329" watchObservedRunningTime="2026-09-05 02:46:28.495263519 +0000 UTC m=+1709.503759719"
Sep 05 02:46:28.510969 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:28.510926    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podStartSLOduration=1.676766135 podStartE2EDuration="21.510911642s" totalImagesPullingTime="15.933274768s" totalInitContainerRuntime="3.900870739s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:46:28.50988687 +0000 UTC m=+1709.518383020" watchObservedRunningTime="2026-09-05 02:46:28.510911642 +0000 UTC m=+1709.519407779"
Sep 05 02:46:29.486943 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:29.486904    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:29.487394 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:29.487015    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:29.489888 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:29.489856    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 05 02:46:29.490019 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:29.489857    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 05 02:46:30.488075 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:30.488027    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 05 02:46:30.488583 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:30.488163    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 05 02:46:35.493798 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:35.493757    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:46:35.494222 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:35.494128    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:46:35.494222 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:35.494178    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 05 02:46:35.494656 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:35.494629    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 05 02:46:45.495026 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:45.494990    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 05 02:46:45.495456 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:45.494981    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 05 02:46:55.494567 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:55.494504    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 05 02:46:55.495064 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:46:55.494702    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 05 02:47:05.494328 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:05.494283    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 05 02:47:05.494773 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:05.494592    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 05 02:47:15.494802 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:15.494759    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 05 02:47:15.495200 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:15.494774    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 05 02:47:25.495254 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:25.495211    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 05 02:47:25.495647 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:25.495215    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 05 02:47:35.494563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:35.494508    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 05 02:47:35.495074 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:35.494682    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 05 02:47:45.495549 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:45.495497    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:47:45.495944 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:45.495704    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:47:45.627424 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:45.627393    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"]
Sep 05 02:47:45.834197 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:45.834076    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container" containerID="cri-o://cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b" gracePeriod=30
Sep 05 02:47:45.834197 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:45.834104    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kube-rbac-proxy" containerID="cri-o://b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda" gracePeriod=30
Sep 05 02:47:46.841100 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:46.841066    2555 generic.go:356] "Generic (PLEG): container finished" podID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerID="b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda" exitCode=2
Sep 05 02:47:46.841463 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:46.841127    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" event={"ID":"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40","Type":"ContainerDied","Data":"b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda"}
Sep 05 02:47:47.550391 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:47.550359    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"]
Sep 05 02:47:47.550648 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:47.550614    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container" containerID="cri-o://03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a" gracePeriod=30
Sep 05 02:47:47.550840 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:47.550682    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kube-rbac-proxy" containerID="cri-o://f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2" gracePeriod=30
Sep 05 02:47:47.847327 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:47.847231    2555 generic.go:356] "Generic (PLEG): container finished" podID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerID="f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2" exitCode=2
Sep 05 02:47:47.847327 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:47.847310    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" event={"ID":"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe","Type":"ContainerDied","Data":"f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2"}
Sep 05 02:47:50.488927 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.488887    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 05 02:47:50.489336 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.488887    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 05 02:47:50.577007 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.576981    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:47:50.671021 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.670985    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kserve-provision-location\") pod \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") "
Sep 05 02:47:50.671021 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.671028    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-proxy-tls\") pod \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") "
Sep 05 02:47:50.671297 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.671047    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9vmf\" (UniqueName: \"kubernetes.io/projected/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-api-access-h9vmf\") pod \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") "
Sep 05 02:47:50.671297 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.671092    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-cabundle-cert\") pod \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") "
Sep 05 02:47:50.671297 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.671202    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-rbac-proxy-sar-config\") pod \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\" (UID: \"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40\") "
Sep 05 02:47:50.671491 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.671460    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" (UID: "5ccc2a4c-6982-41ac-a8f8-171fde8a3b40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:47:50.671576 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.671506    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" (UID: "5ccc2a4c-6982-41ac-a8f8-171fde8a3b40"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:47:50.671625 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.671590    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" (UID: "5ccc2a4c-6982-41ac-a8f8-171fde8a3b40"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:47:50.673154 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.673135    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" (UID: "5ccc2a4c-6982-41ac-a8f8-171fde8a3b40"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:47:50.673219 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.673158    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-api-access-h9vmf" (OuterVolumeSpecName: "kube-api-access-h9vmf") pod "5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" (UID: "5ccc2a4c-6982-41ac-a8f8-171fde8a3b40"). InnerVolumeSpecName "kube-api-access-h9vmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:47:50.772449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.772393    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:47:50.772449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.772432    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:47:50.772449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.772442    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9vmf\" (UniqueName: \"kubernetes.io/projected/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-api-access-h9vmf\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:47:50.772449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.772451    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:47:50.772449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.772460    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:47:50.860839 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.860796    2555 generic.go:356] "Generic (PLEG): container finished" podID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerID="cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b" exitCode=0
Sep 05 02:47:50.860998 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.860861    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"
Sep 05 02:47:50.860998 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.860860    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" event={"ID":"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40","Type":"ContainerDied","Data":"cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b"}
Sep 05 02:47:50.860998 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.860934    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg" event={"ID":"5ccc2a4c-6982-41ac-a8f8-171fde8a3b40","Type":"ContainerDied","Data":"b0a43d88144bfbcf3c3794bb7f79ae026651ced957a2059b5ce2b4edbf0529a9"}
Sep 05 02:47:50.860998 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.860971    2555 scope.go:117] "RemoveContainer" containerID="b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda"
Sep 05 02:47:50.869673 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.869658    2555 scope.go:117] "RemoveContainer" containerID="cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b"
Sep 05 02:47:50.877311 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.877292    2555 scope.go:117] "RemoveContainer" containerID="be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed"
Sep 05 02:47:50.883289 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.883270    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"]
Sep 05 02:47:50.885011 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.884991    2555 scope.go:117] "RemoveContainer" containerID="b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda"
Sep 05 02:47:50.885285 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:47:50.885265    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda\": container with ID starting with b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda not found: ID does not exist" containerID="b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda"
Sep 05 02:47:50.885351 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.885300    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda"} err="failed to get container status \"b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda\": rpc error: code = NotFound desc = could not find container \"b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda\": container with ID starting with b364923bb7389f999f2286287c969cfa7232a7ea7844ada0ee1a187df927eeda not found: ID does not exist"
Sep 05 02:47:50.885351 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.885326    2555 scope.go:117] "RemoveContainer" containerID="cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b"
Sep 05 02:47:50.885605 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:47:50.885588    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b\": container with ID starting with cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b not found: ID does not exist" containerID="cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b"
Sep 05 02:47:50.885670 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.885611    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b"} err="failed to get container status \"cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b\": rpc error: code = NotFound desc = could not find container \"cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b\": container with ID starting with cb5241b240b888cac44f5e845bd306eeba9009e3f86527749e8d1f742d34bb4b not found: ID does not exist"
Sep 05 02:47:50.885670 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.885626    2555 scope.go:117] "RemoveContainer" containerID="be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed"
Sep 05 02:47:50.885860 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:47:50.885842    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed\": container with ID starting with be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed not found: ID does not exist" containerID="be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed"
Sep 05 02:47:50.885916 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.885872    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed"} err="failed to get container status \"be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed\": rpc error: code = NotFound desc = could not find container \"be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed\": container with ID starting with be9008975050239e333d6641a82e6078b7dfc4e831c697f40cf8f92f179afaed not found: ID does not exist"
Sep 05 02:47:50.889734 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:50.889711    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-r68mg"]
Sep 05 02:47:51.543711 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:51.543681    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" path="/var/lib/kubelet/pods/5ccc2a4c-6982-41ac-a8f8-171fde8a3b40/volumes"
Sep 05 02:47:52.497224 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.497201    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:47:52.585449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.585360    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-rbac-proxy-sar-config\") pod \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") "
Sep 05 02:47:52.585449 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.585403    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-cabundle-cert\") pod \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") "
Sep 05 02:47:52.585984 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.585494    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4kk8\" (UniqueName: \"kubernetes.io/projected/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-api-access-r4kk8\") pod \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") "
Sep 05 02:47:52.585984 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.585545    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-proxy-tls\") pod \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") "
Sep 05 02:47:52.585984 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.585610    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kserve-provision-location\") pod \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\" (UID: \"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe\") "
Sep 05 02:47:52.585984 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.585758    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" (UID: "48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:47:52.585984 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.585788    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" (UID: "48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:47:52.585984 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.585957    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" (UID: "48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:47:52.586205 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.586019    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:47:52.586205 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.586030    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:47:52.586205 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.586040    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:47:52.587668 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.587642    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-api-access-r4kk8" (OuterVolumeSpecName: "kube-api-access-r4kk8") pod "48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" (UID: "48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe"). InnerVolumeSpecName "kube-api-access-r4kk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:47:52.587668 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.587665    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" (UID: "48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:47:52.687320 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.687279    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kk8\" (UniqueName: \"kubernetes.io/projected/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-kube-api-access-r4kk8\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:47:52.687320 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.687319    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.718858    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"]
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719642    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kube-rbac-proxy"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719661    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kube-rbac-proxy"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719699    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719707    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719745    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="storage-initializer"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719775    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="storage-initializer"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719786    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="storage-initializer"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719796    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="storage-initializer"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719818    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719827    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719847    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kube-rbac-proxy"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.719856    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kube-rbac-proxy"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.720003    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kube-rbac-proxy"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.720015    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kserve-container"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.720031    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerName="kserve-container"
Sep 05 02:47:52.722131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.720041    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ccc2a4c-6982-41ac-a8f8-171fde8a3b40" containerName="kube-rbac-proxy"
Sep 05 02:47:52.725424 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.725401    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.727825 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.727804    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 05 02:47:52.727942 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.727867    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 05 02:47:52.731089 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.731066    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"]
Sep 05 02:47:52.755600 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.755568    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"]
Sep 05 02:47:52.758953 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.758933    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.761714 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.761698    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 05 02:47:52.773188 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.773165    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"]
Sep 05 02:47:52.787805 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.787783    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.787910 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.787813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjn4\" (UniqueName: \"kubernetes.io/projected/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-api-access-bkjn4\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.787910 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.787832    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.787999 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.787955    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.787999 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.787975    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.788087 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.788049    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.788130 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.788088    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.788130 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.788120    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt59m\" (UniqueName: \"kubernetes.io/projected/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-api-access-wt59m\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.788210 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.788135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.788210 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.788154    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.872834 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.872803    2555 generic.go:356] "Generic (PLEG): container finished" podID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" containerID="03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a" exitCode=0
Sep 05 02:47:52.872987 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.872868    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"
Sep 05 02:47:52.872987 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.872884    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" event={"ID":"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe","Type":"ContainerDied","Data":"03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a"}
Sep 05 02:47:52.872987 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.872945    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk" event={"ID":"48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe","Type":"ContainerDied","Data":"ff83ab09dd5783d73c1baf4f259f027a5cecad648bbd622273936396abed2b11"}
Sep 05 02:47:52.872987 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.872974    2555 scope.go:117] "RemoveContainer" containerID="f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2"
Sep 05 02:47:52.881344 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.881328    2555 scope.go:117] "RemoveContainer" containerID="03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a"
Sep 05 02:47:52.888802 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.888710    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.888802 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.888754    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.888802 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.888788    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.888969 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.888815    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.888969 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.888820    2555 scope.go:117] "RemoveContainer" containerID="dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc"
Sep 05 02:47:52.888969 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.888845    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt59m\" (UniqueName: \"kubernetes.io/projected/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-api-access-wt59m\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.888969 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.888895    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.888969 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.888940    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.889219 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.888988    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.889219 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.889014    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjn4\" (UniqueName: \"kubernetes.io/projected/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-api-access-bkjn4\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.889219 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.889043    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.889219 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.889074    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.889457 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.889435    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.889586 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.889562    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.889724 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.889705    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.889881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.889858    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.890153 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.890127    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.891628 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.891605    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.891781 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.891695    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.894133 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.894109    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"]
Sep 05 02:47:52.896074 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.896018    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt59m\" (UniqueName: \"kubernetes.io/projected/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-api-access-wt59m\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-t7t5n\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:52.897753 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.897734    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjn4\" (UniqueName: \"kubernetes.io/projected/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-api-access-bkjn4\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-t9ds9\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:52.898099 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.898080    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-7thtk"]
Sep 05 02:47:52.903470 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.903451    2555 scope.go:117] "RemoveContainer" containerID="f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2"
Sep 05 02:47:52.903766 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:47:52.903749    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2\": container with ID starting with f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2 not found: ID does not exist" containerID="f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2"
Sep 05 02:47:52.903854 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.903778    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2"} err="failed to get container status \"f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2\": rpc error: code = NotFound desc = could not find container \"f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2\": container with ID starting with f72da57b0c0f9d44ff992c6081f5ee98bb21fe03e75230a687463444e67c6ff2 not found: ID does not exist"
Sep 05 02:47:52.903854 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.903804    2555 scope.go:117] "RemoveContainer" containerID="03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a"
Sep 05 02:47:52.904081 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:47:52.904064    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a\": container with ID starting with 03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a not found: ID does not exist" containerID="03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a"
Sep 05 02:47:52.904128 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.904087    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a"} err="failed to get container status \"03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a\": rpc error: code = NotFound desc = could not find container \"03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a\": container with ID starting with 03d59df296d8d4f46beda0b3ae797b500bd5b736e1ed111d3ee3ed54c92d050a not found: ID does not exist"
Sep 05 02:47:52.904128 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.904101    2555 scope.go:117] "RemoveContainer" containerID="dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc"
Sep 05 02:47:52.904341 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:47:52.904327    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc\": container with ID starting with dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc not found: ID does not exist" containerID="dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc"
Sep 05 02:47:52.904379 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:52.904346    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc"} err="failed to get container status \"dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc\": rpc error: code = NotFound desc = could not find container \"dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc\": container with ID starting with dbe7bad92961dff278c1616ffd0b22ad7d43812d80954691d9e4089964cd20dc not found: ID does not exist"
Sep 05 02:47:53.037629 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:53.037594    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:53.069071 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:53.069041    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:53.310884 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:53.310861    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"]
Sep 05 02:47:53.312737 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:47:53.312707    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92b28d8_a6b3_48f5_bff2_49f9c0f5b35c.slice/crio-0433dd814fa47da1b853cd0788bc3609fb8b6cd964f2bd53909725450ca306eb WatchSource:0}: Error finding container 0433dd814fa47da1b853cd0788bc3609fb8b6cd964f2bd53909725450ca306eb: Status 404 returned error can't find the container with id 0433dd814fa47da1b853cd0788bc3609fb8b6cd964f2bd53909725450ca306eb
Sep 05 02:47:53.345866 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:53.345838    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"]
Sep 05 02:47:53.547801 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:53.547765    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe" path="/var/lib/kubelet/pods/48cb1fb9-da74-4e3c-ba2e-73cd96c5dcbe/volumes"
Sep 05 02:47:53.883006 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:53.882964    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" event={"ID":"92adaf9d-38fb-4003-89b6-28f47a0cd9f2","Type":"ContainerStarted","Data":"35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef"}
Sep 05 02:47:53.883463 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:53.883042    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" event={"ID":"92adaf9d-38fb-4003-89b6-28f47a0cd9f2","Type":"ContainerStarted","Data":"41eb017e34d018f0df4c8ded1be76c9442efd78bf840c719a540423b99f242e7"}
Sep 05 02:47:53.884696 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:53.884662    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" event={"ID":"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c","Type":"ContainerStarted","Data":"acc5b599b085e841d5c5cbb28c4c18883e0f707a21f890af4241db4833afba38"}
Sep 05 02:47:53.884821 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:53.884726    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" event={"ID":"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c","Type":"ContainerStarted","Data":"0433dd814fa47da1b853cd0788bc3609fb8b6cd964f2bd53909725450ca306eb"}
Sep 05 02:47:57.906193 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:57.906158    2555 generic.go:356] "Generic (PLEG): container finished" podID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerID="35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef" exitCode=0
Sep 05 02:47:57.906634 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:57.906212    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" event={"ID":"92adaf9d-38fb-4003-89b6-28f47a0cd9f2","Type":"ContainerDied","Data":"35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef"}
Sep 05 02:47:57.908011 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:57.907990    2555 generic.go:356] "Generic (PLEG): container finished" podID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerID="acc5b599b085e841d5c5cbb28c4c18883e0f707a21f890af4241db4833afba38" exitCode=0
Sep 05 02:47:57.908115 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:57.908064    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" event={"ID":"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c","Type":"ContainerDied","Data":"acc5b599b085e841d5c5cbb28c4c18883e0f707a21f890af4241db4833afba38"}
Sep 05 02:47:58.915145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:58.915105    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" event={"ID":"92adaf9d-38fb-4003-89b6-28f47a0cd9f2","Type":"ContainerStarted","Data":"a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf"}
Sep 05 02:47:58.915603 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:58.915175    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" event={"ID":"92adaf9d-38fb-4003-89b6-28f47a0cd9f2","Type":"ContainerStarted","Data":"c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8"}
Sep 05 02:47:58.915603 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:58.915398    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:58.917287 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:58.917264    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" event={"ID":"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c","Type":"ContainerStarted","Data":"6088c605eba8bdcc77cfe7e0abafc48b5ea7aedad98afcb7f3d9017e8fc735b1"}
Sep 05 02:47:58.917411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:58.917308    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" event={"ID":"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c","Type":"ContainerStarted","Data":"d20ebd88b20bdc44d424d45629e25723894cf6277c91c7cdb653cc4a42aefc0e"}
Sep 05 02:47:58.917572 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:58.917556    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:58.917636 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:58.917603    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:47:58.919112 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:58.919088    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 05 02:47:58.937617 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:58.937571    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podStartSLOduration=3.244660211 podStartE2EDuration="6.937554253s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.692894042s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:47:58.935263967 +0000 UTC m=+1799.943760106" watchObservedRunningTime="2026-09-05 02:47:58.937554253 +0000 UTC m=+1799.946050455"
Sep 05 02:47:58.958984 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:58.958936    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podStartSLOduration=3.047706985 podStartE2EDuration="6.958918109s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.911211124s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:47:58.957356201 +0000 UTC m=+1799.965852362" watchObservedRunningTime="2026-09-05 02:47:58.958918109 +0000 UTC m=+1799.967414313"
Sep 05 02:47:59.583118 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:59.583091    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:47:59.584415 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:59.584389    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:47:59.922306 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:59.922258    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 05 02:47:59.922772 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:59.922569    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:47:59.924068 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:47:59.924032    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 05 02:48:00.926930 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:00.926893    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 05 02:48:04.926785 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:04.926754    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:48:04.927293 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:04.927263    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 05 02:48:05.931795 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:05.931766    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:48:05.932354 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:05.932319    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 05 02:48:14.927364 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:14.927321    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 05 02:48:15.932761 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:15.932715    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 05 02:48:24.928023 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:24.927977    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 05 02:48:25.932526 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:25.932483    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 05 02:48:34.927574 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:34.927520    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 05 02:48:35.932450 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:35.932410    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 05 02:48:44.927502 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:44.927460    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 05 02:48:45.932290 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:45.932245    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 05 02:48:54.927582 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:54.927525    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 05 02:48:55.933099 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:48:55.933061    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 05 02:49:04.927775 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:04.927734    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 05 02:49:05.933126 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:05.933078    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 05 02:49:06.539389 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:06.539341    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 05 02:49:14.928240 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:14.928202    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:49:15.076636 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:15.076603    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"]
Sep 05 02:49:15.076924 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:15.076892    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container" containerID="cri-o://c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8" gracePeriod=30
Sep 05 02:49:15.077048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:15.076943    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kube-rbac-proxy" containerID="cri-o://a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf" gracePeriod=30
Sep 05 02:49:15.078212 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:15.078192    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:49:15.274009 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:15.273977    2555 generic.go:356] "Generic (PLEG): container finished" podID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerID="a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf" exitCode=2
Sep 05 02:49:15.274171 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:15.274032    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" event={"ID":"92adaf9d-38fb-4003-89b6-28f47a0cd9f2","Type":"ContainerDied","Data":"a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf"}
Sep 05 02:49:15.928080 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:15.928041    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 05 02:49:19.826982 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.826952    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:49:19.857597 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.857568    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kserve-provision-location\") pod \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") "
Sep 05 02:49:19.857764 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.857622    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-proxy-tls\") pod \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") "
Sep 05 02:49:19.857764 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.857658    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-rbac-proxy-sar-config\") pod \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") "
Sep 05 02:49:19.857764 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.857682    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-cabundle-cert\") pod \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") "
Sep 05 02:49:19.857764 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.857720    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkjn4\" (UniqueName: \"kubernetes.io/projected/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-api-access-bkjn4\") pod \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\" (UID: \"92adaf9d-38fb-4003-89b6-28f47a0cd9f2\") "
Sep 05 02:49:19.858042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.858008    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92adaf9d-38fb-4003-89b6-28f47a0cd9f2" (UID: "92adaf9d-38fb-4003-89b6-28f47a0cd9f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:49:19.858042 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.858029    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "92adaf9d-38fb-4003-89b6-28f47a0cd9f2" (UID: "92adaf9d-38fb-4003-89b6-28f47a0cd9f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:49:19.858181 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.858044    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92adaf9d-38fb-4003-89b6-28f47a0cd9f2" (UID: "92adaf9d-38fb-4003-89b6-28f47a0cd9f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:49:19.859876 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.859844    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92adaf9d-38fb-4003-89b6-28f47a0cd9f2" (UID: "92adaf9d-38fb-4003-89b6-28f47a0cd9f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:49:19.859992 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.859902    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-api-access-bkjn4" (OuterVolumeSpecName: "kube-api-access-bkjn4") pod "92adaf9d-38fb-4003-89b6-28f47a0cd9f2" (UID: "92adaf9d-38fb-4003-89b6-28f47a0cd9f2"). InnerVolumeSpecName "kube-api-access-bkjn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:49:19.958380 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.958298    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:49:19.958380 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.958327    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:49:19.958380 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.958337    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:49:19.958380 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.958346    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkjn4\" (UniqueName: \"kubernetes.io/projected/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kube-api-access-bkjn4\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:49:19.958380 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:19.958357    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92adaf9d-38fb-4003-89b6-28f47a0cd9f2-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:49:20.298477 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.298388    2555 generic.go:356] "Generic (PLEG): container finished" podID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerID="c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8" exitCode=0
Sep 05 02:49:20.298477 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.298441    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"
Sep 05 02:49:20.298712 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.298472    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" event={"ID":"92adaf9d-38fb-4003-89b6-28f47a0cd9f2","Type":"ContainerDied","Data":"c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8"}
Sep 05 02:49:20.298712 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.298529    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9" event={"ID":"92adaf9d-38fb-4003-89b6-28f47a0cd9f2","Type":"ContainerDied","Data":"41eb017e34d018f0df4c8ded1be76c9442efd78bf840c719a540423b99f242e7"}
Sep 05 02:49:20.298712 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.298558    2555 scope.go:117] "RemoveContainer" containerID="a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf"
Sep 05 02:49:20.307395 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.307379    2555 scope.go:117] "RemoveContainer" containerID="c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8"
Sep 05 02:49:20.314711 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.314694    2555 scope.go:117] "RemoveContainer" containerID="35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef"
Sep 05 02:49:20.322385 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.322350    2555 scope.go:117] "RemoveContainer" containerID="a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf"
Sep 05 02:49:20.322655 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:49:20.322638    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf\": container with ID starting with a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf not found: ID does not exist" containerID="a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf"
Sep 05 02:49:20.322724 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.322662    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf"} err="failed to get container status \"a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf\": rpc error: code = NotFound desc = could not find container \"a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf\": container with ID starting with a7ea1156b1beadc752e5b4c752678d012f2d337c5d386b5aef2a382094571dbf not found: ID does not exist"
Sep 05 02:49:20.322724 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.322679    2555 scope.go:117] "RemoveContainer" containerID="c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8"
Sep 05 02:49:20.322989 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:49:20.322962    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8\": container with ID starting with c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8 not found: ID does not exist" containerID="c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8"
Sep 05 02:49:20.323038 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.322998    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8"} err="failed to get container status \"c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8\": rpc error: code = NotFound desc = could not find container \"c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8\": container with ID starting with c9ecf4dfdc6326cb610391affa6700a6f3273d6a5e730d09c257a88ffb1967b8 not found: ID does not exist"
Sep 05 02:49:20.323038 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.323020    2555 scope.go:117] "RemoveContainer" containerID="35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef"
Sep 05 02:49:20.323276 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:49:20.323257    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef\": container with ID starting with 35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef not found: ID does not exist" containerID="35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef"
Sep 05 02:49:20.323331 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.323281    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef"} err="failed to get container status \"35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef\": rpc error: code = NotFound desc = could not find container \"35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef\": container with ID starting with 35ca570d37f1811c80e8b937cfa9280f82979f2530951367e46611531b3ae4ef not found: ID does not exist"
Sep 05 02:49:20.324351 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.324328    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"]
Sep 05 02:49:20.325873 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:20.325856    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-t9ds9"]
Sep 05 02:49:21.544138 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:49:21.544106    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" path="/var/lib/kubelet/pods/92adaf9d-38fb-4003-89b6-28f47a0cd9f2/volumes"
Sep 05 02:50:21.301231 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.301192    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"]
Sep 05 02:50:21.301683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.301566    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container"
Sep 05 02:50:21.301683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.301578    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container"
Sep 05 02:50:21.301683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.301590    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="storage-initializer"
Sep 05 02:50:21.301683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.301596    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="storage-initializer"
Sep 05 02:50:21.301683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.301611    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kube-rbac-proxy"
Sep 05 02:50:21.301683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.301617    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kube-rbac-proxy"
Sep 05 02:50:21.301683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.301686    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kube-rbac-proxy"
Sep 05 02:50:21.301912 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.301692    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="92adaf9d-38fb-4003-89b6-28f47a0cd9f2" containerName="kserve-container"
Sep 05 02:50:21.304858 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.304840    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.308055 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.308027    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 05 02:50:21.308192 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.308076    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 05 02:50:21.319666 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.319638    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"]
Sep 05 02:50:21.381795 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.381754    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6khc\" (UniqueName: \"kubernetes.io/projected/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-api-access-x6khc\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.381795 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.381793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.382014 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.381858    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.382014 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.381879    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc4a057f-325a-451f-8da9-a8eaff25a35b-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.382014 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.381960    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc4a057f-325a-451f-8da9-a8eaff25a35b-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.483182 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.483142    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc4a057f-325a-451f-8da9-a8eaff25a35b-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.483368 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.483226    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc4a057f-325a-451f-8da9-a8eaff25a35b-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.483368 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.483285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6khc\" (UniqueName: \"kubernetes.io/projected/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-api-access-x6khc\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.483492 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.483474    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.483588 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.483570    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.483755 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.483728    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc4a057f-325a-451f-8da9-a8eaff25a35b-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.484143 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.484123    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.484181 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.484143    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.485837 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.485819    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc4a057f-325a-451f-8da9-a8eaff25a35b-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.491915 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.491890    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6khc\" (UniqueName: \"kubernetes.io/projected/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-api-access-x6khc\") pod \"isvc-tensorflow-predictor-547dfc5669-w5v6x\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.615330 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.615219    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:21.890753 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:21.890728    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"]
Sep 05 02:50:21.893385 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:50:21.893354    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4a057f_325a_451f_8da9_a8eaff25a35b.slice/crio-606d3062563ee45e19cd70a255fb4e778a92a776887af857979e214cb7e6a249 WatchSource:0}: Error finding container 606d3062563ee45e19cd70a255fb4e778a92a776887af857979e214cb7e6a249: Status 404 returned error can't find the container with id 606d3062563ee45e19cd70a255fb4e778a92a776887af857979e214cb7e6a249
Sep 05 02:50:22.579040 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:22.578995    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" event={"ID":"cc4a057f-325a-451f-8da9-a8eaff25a35b","Type":"ContainerStarted","Data":"ec3c75bb854938c0f75d48cab0fc2b5e0458d30eed9bb5e464882edc2ad4daf9"}
Sep 05 02:50:22.579524 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:22.579068    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" event={"ID":"cc4a057f-325a-451f-8da9-a8eaff25a35b","Type":"ContainerStarted","Data":"606d3062563ee45e19cd70a255fb4e778a92a776887af857979e214cb7e6a249"}
Sep 05 02:50:27.611131 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:27.611093    2555 generic.go:356] "Generic (PLEG): container finished" podID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerID="ec3c75bb854938c0f75d48cab0fc2b5e0458d30eed9bb5e464882edc2ad4daf9" exitCode=0
Sep 05 02:50:27.611523 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:27.611160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" event={"ID":"cc4a057f-325a-451f-8da9-a8eaff25a35b","Type":"ContainerDied","Data":"ec3c75bb854938c0f75d48cab0fc2b5e0458d30eed9bb5e464882edc2ad4daf9"}
Sep 05 02:50:31.635880 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:31.635844    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" event={"ID":"cc4a057f-325a-451f-8da9-a8eaff25a35b","Type":"ContainerStarted","Data":"f221030bc1066d3b265a7c42cf4cf5b01e4fd1d0fd75140a0b38163a19bfda6f"}
Sep 05 02:50:31.636336 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:31.635910    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" event={"ID":"cc4a057f-325a-451f-8da9-a8eaff25a35b","Type":"ContainerStarted","Data":"62026420afef2e296ceedaf85649d5a9e8315475e9b2620366301dd3b4dc3825"}
Sep 05 02:50:31.636336 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:31.636111    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:31.656692 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:31.656582    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podStartSLOduration=2.229646055 podStartE2EDuration="10.656562083s" totalImagesPullingTime="3.780792111s" totalInitContainerRuntime="4.646123917s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:50:31.654511629 +0000 UTC m=+1952.663007789" watchObservedRunningTime="2026-09-05 02:50:31.656562083 +0000 UTC m=+1952.665058219"
Sep 05 02:50:32.640772 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:32.640732    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:32.642092 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:32.642063    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 05 02:50:33.645476 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:33.645433    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 05 02:50:38.650678 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:38.650645    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:38.651264 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:38.651237    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 05 02:50:48.652072 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:48.652041    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:50:52.961128 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:52.961043    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"]
Sep 05 02:50:52.961606 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:52.961363    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container" containerID="cri-o://d20ebd88b20bdc44d424d45629e25723894cf6277c91c7cdb653cc4a42aefc0e" gracePeriod=30
Sep 05 02:50:52.961606 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:52.961449    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kube-rbac-proxy" containerID="cri-o://6088c605eba8bdcc77cfe7e0abafc48b5ea7aedad98afcb7f3d9017e8fc735b1" gracePeriod=30
Sep 05 02:50:53.733975 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:53.733937    2555 generic.go:356] "Generic (PLEG): container finished" podID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerID="6088c605eba8bdcc77cfe7e0abafc48b5ea7aedad98afcb7f3d9017e8fc735b1" exitCode=2
Sep 05 02:50:53.734192 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:53.734002    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" event={"ID":"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c","Type":"ContainerDied","Data":"6088c605eba8bdcc77cfe7e0abafc48b5ea7aedad98afcb7f3d9017e8fc735b1"}
Sep 05 02:50:54.923420 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:54.923376    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 05 02:50:54.927724 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:54.927692    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 05 02:50:57.759102 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.759062    2555 generic.go:356] "Generic (PLEG): container finished" podID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerID="d20ebd88b20bdc44d424d45629e25723894cf6277c91c7cdb653cc4a42aefc0e" exitCode=0
Sep 05 02:50:57.759102 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.759103    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" event={"ID":"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c","Type":"ContainerDied","Data":"d20ebd88b20bdc44d424d45629e25723894cf6277c91c7cdb653cc4a42aefc0e"}
Sep 05 02:50:57.798574 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.798551    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:50:57.898407 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.898316    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-proxy-tls\") pod \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") "
Sep 05 02:50:57.898407 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.898364    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt59m\" (UniqueName: \"kubernetes.io/projected/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-api-access-wt59m\") pod \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") "
Sep 05 02:50:57.898685 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.898426    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-rbac-proxy-sar-config\") pod \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") "
Sep 05 02:50:57.898685 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.898448    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-cabundle-cert\") pod \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") "
Sep 05 02:50:57.898685 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.898593    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kserve-provision-location\") pod \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\" (UID: \"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c\") "
Sep 05 02:50:57.898913 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.898888    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" (UID: "e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:50:57.898970 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.898904    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" (UID: "e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:57.898970 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.898915    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" (UID: "e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:50:57.900485 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.900468    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-api-access-wt59m" (OuterVolumeSpecName: "kube-api-access-wt59m") pod "e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" (UID: "e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c"). InnerVolumeSpecName "kube-api-access-wt59m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:50:57.900628 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:57.900607    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" (UID: "e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:50:58.000168 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.000133    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:50:58.000168 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.000164    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:50:58.000168 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.000175    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:50:58.000429 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.000186    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:50:58.000429 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.000194    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt59m\" (UniqueName: \"kubernetes.io/projected/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c-kube-api-access-wt59m\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:50:58.149663 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.149572    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"]
Sep 05 02:50:58.150003 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.149986    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="storage-initializer"
Sep 05 02:50:58.150086 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.150006    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="storage-initializer"
Sep 05 02:50:58.150086 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.150032    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container"
Sep 05 02:50:58.150086 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.150042    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container"
Sep 05 02:50:58.150086 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.150061    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kube-rbac-proxy"
Sep 05 02:50:58.150086 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.150071    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kube-rbac-proxy"
Sep 05 02:50:58.150348 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.150165    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kube-rbac-proxy"
Sep 05 02:50:58.150348 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.150178    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" containerName="kserve-container"
Sep 05 02:50:58.153587 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.153564    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.156704 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.156682    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 05 02:50:58.156704 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.156697    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 05 02:50:58.164155 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.164129    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"]
Sep 05 02:50:58.243112 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.243073    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"]
Sep 05 02:50:58.246760 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.246738    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.249058 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.249039    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 05 02:50:58.256468 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.256445    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"]
Sep 05 02:50:58.303189 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.303152    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/560f41d1-d471-47a7-9f9b-00ecfc28ea39-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.303189 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.303191    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfwm\" (UniqueName: \"kubernetes.io/projected/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-api-access-wlfwm\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.303424 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.303299    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.303424 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.303341    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.303424 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.303405    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.404048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.403952    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.404048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.404010    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.404294 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.404080    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.404294 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.404111    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.404294 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.404139    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.404294 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.404172    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.404294 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.404215    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.404294 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.404250    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whs57\" (UniqueName: \"kubernetes.io/projected/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-api-access-whs57\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.404294 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.404282    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/560f41d1-d471-47a7-9f9b-00ecfc28ea39-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.404672 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.404308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfwm\" (UniqueName: \"kubernetes.io/projected/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-api-access-wlfwm\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.407277 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.405016    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.407277 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.405256    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.407277 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.405267    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.411832 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.411804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/560f41d1-d471-47a7-9f9b-00ecfc28ea39-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.413977 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.413945    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfwm\" (UniqueName: \"kubernetes.io/projected/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-api-access-wlfwm\") pod \"isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.463863 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.463827    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:50:58.505332 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.505289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whs57\" (UniqueName: \"kubernetes.io/projected/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-api-access-whs57\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.505527 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.505344    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.505527 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.505389    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.505527 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.505442    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.505527 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.505465    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.505961 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.505931    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.506298 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.506276    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.506369 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.506313    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.508002 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.507980    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-proxy-tls\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.513917 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.513891    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whs57\" (UniqueName: \"kubernetes.io/projected/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-api-access-whs57\") pod \"isvc-canary-rollback-predictor-76998db6fd-42sq2\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.557671 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.557620    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:50:58.737514 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.737486    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"]
Sep 05 02:50:58.739849 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:50:58.739809    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560f41d1_d471_47a7_9f9b_00ecfc28ea39.slice/crio-92a08b36ce2d3116dd2af932545a3f181d904033aeb0f3050e7fbd488a725e7c WatchSource:0}: Error finding container 92a08b36ce2d3116dd2af932545a3f181d904033aeb0f3050e7fbd488a725e7c: Status 404 returned error can't find the container with id 92a08b36ce2d3116dd2af932545a3f181d904033aeb0f3050e7fbd488a725e7c
Sep 05 02:50:58.765168 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.765134    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" event={"ID":"560f41d1-d471-47a7-9f9b-00ecfc28ea39","Type":"ContainerStarted","Data":"92a08b36ce2d3116dd2af932545a3f181d904033aeb0f3050e7fbd488a725e7c"}
Sep 05 02:50:58.767226 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.767204    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"
Sep 05 02:50:58.767322 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.767203    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n" event={"ID":"e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c","Type":"ContainerDied","Data":"0433dd814fa47da1b853cd0788bc3609fb8b6cd964f2bd53909725450ca306eb"}
Sep 05 02:50:58.767378 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.767339    2555 scope.go:117] "RemoveContainer" containerID="6088c605eba8bdcc77cfe7e0abafc48b5ea7aedad98afcb7f3d9017e8fc735b1"
Sep 05 02:50:58.792463 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.792424    2555 scope.go:117] "RemoveContainer" containerID="d20ebd88b20bdc44d424d45629e25723894cf6277c91c7cdb653cc4a42aefc0e"
Sep 05 02:50:58.804031 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.803995    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"]
Sep 05 02:50:58.807268 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.807243    2555 scope.go:117] "RemoveContainer" containerID="acc5b599b085e841d5c5cbb28c4c18883e0f707a21f890af4241db4833afba38"
Sep 05 02:50:58.809332 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.809309    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-t7t5n"]
Sep 05 02:50:58.830308 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:58.830275    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"]
Sep 05 02:50:58.832809 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:50:58.832772    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e9b29a_e3b3_4efd_8ca9_365972d9b38e.slice/crio-00f3fbf891535b5934c6c28b7a16c8aef34dac4bdb5b467aad6e0147992b49a1 WatchSource:0}: Error finding container 00f3fbf891535b5934c6c28b7a16c8aef34dac4bdb5b467aad6e0147992b49a1: Status 404 returned error can't find the container with id 00f3fbf891535b5934c6c28b7a16c8aef34dac4bdb5b467aad6e0147992b49a1
Sep 05 02:50:59.547404 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:59.547370    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c" path="/var/lib/kubelet/pods/e92b28d8-a6b3-48f5-bff2-49f9c0f5b35c/volumes"
Sep 05 02:50:59.773886 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:59.773847    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" event={"ID":"560f41d1-d471-47a7-9f9b-00ecfc28ea39","Type":"ContainerStarted","Data":"7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1"}
Sep 05 02:50:59.775457 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:59.775433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" event={"ID":"33e9b29a-e3b3-4efd-8ca9-365972d9b38e","Type":"ContainerStarted","Data":"2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161"}
Sep 05 02:50:59.775598 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:50:59.775477    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" event={"ID":"33e9b29a-e3b3-4efd-8ca9-365972d9b38e","Type":"ContainerStarted","Data":"00f3fbf891535b5934c6c28b7a16c8aef34dac4bdb5b467aad6e0147992b49a1"}
Sep 05 02:51:02.462232 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:02.462197    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"]
Sep 05 02:51:02.462684 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:02.462492    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kserve-container" containerID="cri-o://62026420afef2e296ceedaf85649d5a9e8315475e9b2620366301dd3b4dc3825" gracePeriod=30
Sep 05 02:51:02.462684 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:02.462513    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kube-rbac-proxy" containerID="cri-o://f221030bc1066d3b265a7c42cf4cf5b01e4fd1d0fd75140a0b38163a19bfda6f" gracePeriod=30
Sep 05 02:51:02.791098 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:02.791004    2555 generic.go:356] "Generic (PLEG): container finished" podID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerID="7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1" exitCode=0
Sep 05 02:51:02.791098 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:02.791077    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" event={"ID":"560f41d1-d471-47a7-9f9b-00ecfc28ea39","Type":"ContainerDied","Data":"7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1"}
Sep 05 02:51:02.792733 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:02.792708    2555 generic.go:356] "Generic (PLEG): container finished" podID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerID="2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161" exitCode=0
Sep 05 02:51:02.792826 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:02.792777    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" event={"ID":"33e9b29a-e3b3-4efd-8ca9-365972d9b38e","Type":"ContainerDied","Data":"2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161"}
Sep 05 02:51:02.795135 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:02.795108    2555 generic.go:356] "Generic (PLEG): container finished" podID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerID="f221030bc1066d3b265a7c42cf4cf5b01e4fd1d0fd75140a0b38163a19bfda6f" exitCode=2
Sep 05 02:51:02.795208 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:02.795186    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" event={"ID":"cc4a057f-325a-451f-8da9-a8eaff25a35b","Type":"ContainerDied","Data":"f221030bc1066d3b265a7c42cf4cf5b01e4fd1d0fd75140a0b38163a19bfda6f"}
Sep 05 02:51:03.646609 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.646561    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 05 02:51:03.802649 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.802606    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" event={"ID":"560f41d1-d471-47a7-9f9b-00ecfc28ea39","Type":"ContainerStarted","Data":"985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76"}
Sep 05 02:51:03.802848 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.802680    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" event={"ID":"560f41d1-d471-47a7-9f9b-00ecfc28ea39","Type":"ContainerStarted","Data":"c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6"}
Sep 05 02:51:03.803002 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.802979    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:51:03.803173 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.803140    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:51:03.804852 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.804824    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 05 02:51:03.804852 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.804835    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" event={"ID":"33e9b29a-e3b3-4efd-8ca9-365972d9b38e","Type":"ContainerStarted","Data":"6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae"}
Sep 05 02:51:03.804995 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.804883    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" event={"ID":"33e9b29a-e3b3-4efd-8ca9-365972d9b38e","Type":"ContainerStarted","Data":"afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3"}
Sep 05 02:51:03.805168 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.805145    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:51:03.805269 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.805187    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:51:03.806179 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.806158    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 05 02:51:03.823050 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.823006    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podStartSLOduration=2.085777049 podStartE2EDuration="5.822973787s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.737196738s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:51:03.821961092 +0000 UTC m=+1984.830457246" watchObservedRunningTime="2026-09-05 02:51:03.822973787 +0000 UTC m=+1984.831469989"
Sep 05 02:51:03.840858 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:03.840809    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podStartSLOduration=2.036972371 podStartE2EDuration="5.840793409s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.803821038s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:51:03.838622737 +0000 UTC m=+1984.847118872" watchObservedRunningTime="2026-09-05 02:51:03.840793409 +0000 UTC m=+1984.849289547"
Sep 05 02:51:04.810458 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:04.810419    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 05 02:51:04.810884 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:04.810590    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 05 02:51:08.646331 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:08.646281    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 05 02:51:09.815162 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:09.815131    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:51:09.815565 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:09.815296    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:51:09.815565 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:09.815494    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 05 02:51:09.816087 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:09.816063    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 05 02:51:13.646455 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:13.646411    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 05 02:51:13.646846 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:13.646544    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:51:18.646021 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:18.645975    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 05 02:51:19.816096 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:19.816055    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 05 02:51:19.816096 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:19.816073    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 05 02:51:23.646183 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:23.646134    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 05 02:51:28.646520 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:28.646462    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 05 02:51:29.815805 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:29.815768    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 05 02:51:29.816188 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:29.815985    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 05 02:51:32.942390 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:32.942354    2555 generic.go:356] "Generic (PLEG): container finished" podID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerID="62026420afef2e296ceedaf85649d5a9e8315475e9b2620366301dd3b4dc3825" exitCode=137
Sep 05 02:51:32.942799 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:32.942425    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" event={"ID":"cc4a057f-325a-451f-8da9-a8eaff25a35b","Type":"ContainerDied","Data":"62026420afef2e296ceedaf85649d5a9e8315475e9b2620366301dd3b4dc3825"}
Sep 05 02:51:33.106108 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.106081    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:51:33.213238 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.213124    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc4a057f-325a-451f-8da9-a8eaff25a35b-kserve-provision-location\") pod \"cc4a057f-325a-451f-8da9-a8eaff25a35b\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") "
Sep 05 02:51:33.213238 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.213201    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6khc\" (UniqueName: \"kubernetes.io/projected/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-api-access-x6khc\") pod \"cc4a057f-325a-451f-8da9-a8eaff25a35b\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") "
Sep 05 02:51:33.213238 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.213220    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-rbac-proxy-sar-config\") pod \"cc4a057f-325a-451f-8da9-a8eaff25a35b\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") "
Sep 05 02:51:33.213238 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.213249    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc4a057f-325a-451f-8da9-a8eaff25a35b-proxy-tls\") pod \"cc4a057f-325a-451f-8da9-a8eaff25a35b\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") "
Sep 05 02:51:33.213632 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.213277    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-cabundle-cert\") pod \"cc4a057f-325a-451f-8da9-a8eaff25a35b\" (UID: \"cc4a057f-325a-451f-8da9-a8eaff25a35b\") "
Sep 05 02:51:33.213697 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.213668    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cc4a057f-325a-451f-8da9-a8eaff25a35b" (UID: "cc4a057f-325a-451f-8da9-a8eaff25a35b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:51:33.213750 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.213715    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cc4a057f-325a-451f-8da9-a8eaff25a35b" (UID: "cc4a057f-325a-451f-8da9-a8eaff25a35b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:51:33.215487 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.215463    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc4a057f-325a-451f-8da9-a8eaff25a35b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc4a057f-325a-451f-8da9-a8eaff25a35b" (UID: "cc4a057f-325a-451f-8da9-a8eaff25a35b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:51:33.215487 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.215464    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-api-access-x6khc" (OuterVolumeSpecName: "kube-api-access-x6khc") pod "cc4a057f-325a-451f-8da9-a8eaff25a35b" (UID: "cc4a057f-325a-451f-8da9-a8eaff25a35b"). InnerVolumeSpecName "kube-api-access-x6khc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:51:33.223960 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.223933    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4a057f-325a-451f-8da9-a8eaff25a35b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc4a057f-325a-451f-8da9-a8eaff25a35b" (UID: "cc4a057f-325a-451f-8da9-a8eaff25a35b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:51:33.314365 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.314333    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc4a057f-325a-451f-8da9-a8eaff25a35b-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:51:33.314365 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.314360    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6khc\" (UniqueName: \"kubernetes.io/projected/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-api-access-x6khc\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:51:33.314365 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.314369    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:51:33.314619 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.314380    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc4a057f-325a-451f-8da9-a8eaff25a35b-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:51:33.314619 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.314389    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc4a057f-325a-451f-8da9-a8eaff25a35b-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:51:33.949586 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.949551    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x" event={"ID":"cc4a057f-325a-451f-8da9-a8eaff25a35b","Type":"ContainerDied","Data":"606d3062563ee45e19cd70a255fb4e778a92a776887af857979e214cb7e6a249"}
Sep 05 02:51:33.950033 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.949627    2555 scope.go:117] "RemoveContainer" containerID="f221030bc1066d3b265a7c42cf4cf5b01e4fd1d0fd75140a0b38163a19bfda6f"
Sep 05 02:51:33.950033 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.949563    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"
Sep 05 02:51:33.958299 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.958278    2555 scope.go:117] "RemoveContainer" containerID="62026420afef2e296ceedaf85649d5a9e8315475e9b2620366301dd3b4dc3825"
Sep 05 02:51:33.965990 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.965969    2555 scope.go:117] "RemoveContainer" containerID="ec3c75bb854938c0f75d48cab0fc2b5e0458d30eed9bb5e464882edc2ad4daf9"
Sep 05 02:51:33.968742 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.968720    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"]
Sep 05 02:51:33.972025 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:33.972004    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-w5v6x"]
Sep 05 02:51:35.543695 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:35.543659    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" path="/var/lib/kubelet/pods/cc4a057f-325a-451f-8da9-a8eaff25a35b/volumes"
Sep 05 02:51:39.815758 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:39.815720    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 05 02:51:39.816170 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:39.815960    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 05 02:51:49.815905 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:49.815861    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 05 02:51:49.816280 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:49.816026    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 05 02:51:59.815676 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:59.815632    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 05 02:51:59.816069 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:51:59.815955    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 05 02:52:09.815738 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:09.815691    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 05 02:52:09.816193 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:09.815958    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 05 02:52:17.538550 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:17.538496    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 05 02:52:19.817244 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:19.817215    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:52:27.545104 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:27.545033    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:52:28.142102 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:28.142058    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"]
Sep 05 02:52:28.216007 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:28.215965    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container" containerID="cri-o://c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6" gracePeriod=30
Sep 05 02:52:28.216227 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:28.216051    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kube-rbac-proxy" containerID="cri-o://985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76" gracePeriod=30
Sep 05 02:52:29.222066 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:29.222032    2555 generic.go:356] "Generic (PLEG): container finished" podID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerID="985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76" exitCode=2
Sep 05 02:52:29.222434 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:29.222104    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" event={"ID":"560f41d1-d471-47a7-9f9b-00ecfc28ea39","Type":"ContainerDied","Data":"985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76"}
Sep 05 02:52:29.810756 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:29.810712    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 05 02:52:33.159603 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.159577    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:52:33.243285 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.243192    2555 generic.go:356] "Generic (PLEG): container finished" podID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerID="c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6" exitCode=0
Sep 05 02:52:33.243285 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.243248    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"
Sep 05 02:52:33.243285 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.243260    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" event={"ID":"560f41d1-d471-47a7-9f9b-00ecfc28ea39","Type":"ContainerDied","Data":"c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6"}
Sep 05 02:52:33.243504 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.243321    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks" event={"ID":"560f41d1-d471-47a7-9f9b-00ecfc28ea39","Type":"ContainerDied","Data":"92a08b36ce2d3116dd2af932545a3f181d904033aeb0f3050e7fbd488a725e7c"}
Sep 05 02:52:33.243504 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.243350    2555 scope.go:117] "RemoveContainer" containerID="985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76"
Sep 05 02:52:33.251762 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.251743    2555 scope.go:117] "RemoveContainer" containerID="c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6"
Sep 05 02:52:33.259559 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.259297    2555 scope.go:117] "RemoveContainer" containerID="7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1"
Sep 05 02:52:33.269103 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.269082    2555 scope.go:117] "RemoveContainer" containerID="985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76"
Sep 05 02:52:33.269392 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:52:33.269361    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76\": container with ID starting with 985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76 not found: ID does not exist" containerID="985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76"
Sep 05 02:52:33.269451 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.269390    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76"} err="failed to get container status \"985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76\": rpc error: code = NotFound desc = could not find container \"985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76\": container with ID starting with 985814750c325a9155b175ffbf44821e337f2daca717df586fca6d48f0b14e76 not found: ID does not exist"
Sep 05 02:52:33.269451 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.269407    2555 scope.go:117] "RemoveContainer" containerID="c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6"
Sep 05 02:52:33.269636 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:52:33.269622    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6\": container with ID starting with c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6 not found: ID does not exist" containerID="c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6"
Sep 05 02:52:33.269681 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.269639    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6"} err="failed to get container status \"c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6\": rpc error: code = NotFound desc = could not find container \"c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6\": container with ID starting with c94b5b4a89f22c3c6d84f78ceb6264fead709aa5fd53088955da401ec9e525f6 not found: ID does not exist"
Sep 05 02:52:33.269681 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.269652    2555 scope.go:117] "RemoveContainer" containerID="7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1"
Sep 05 02:52:33.269900 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:52:33.269884    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1\": container with ID starting with 7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1 not found: ID does not exist" containerID="7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1"
Sep 05 02:52:33.269938 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.269906    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1"} err="failed to get container status \"7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1\": rpc error: code = NotFound desc = could not find container \"7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1\": container with ID starting with 7dae5330a5a2aea25ea695894c12b95ae8e1b17a32959cdd2aa3f4512ae660a1 not found: ID does not exist"
Sep 05 02:52:33.338189 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.338158    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kserve-provision-location\") pod \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") "
Sep 05 02:52:33.338359 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.338198    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-rbac-proxy-sar-config\") pod \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") "
Sep 05 02:52:33.338359 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.338237    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/560f41d1-d471-47a7-9f9b-00ecfc28ea39-proxy-tls\") pod \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") "
Sep 05 02:52:33.338359 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.338261    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlfwm\" (UniqueName: \"kubernetes.io/projected/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-api-access-wlfwm\") pod \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") "
Sep 05 02:52:33.338359 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.338313    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-cabundle-cert\") pod \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\" (UID: \"560f41d1-d471-47a7-9f9b-00ecfc28ea39\") "
Sep 05 02:52:33.338618 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.338467    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "560f41d1-d471-47a7-9f9b-00ecfc28ea39" (UID: "560f41d1-d471-47a7-9f9b-00ecfc28ea39"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:52:33.338679 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.338573    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:52:33.338679 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.338644    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "560f41d1-d471-47a7-9f9b-00ecfc28ea39" (UID: "560f41d1-d471-47a7-9f9b-00ecfc28ea39"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:52:33.338780 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.338735    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "560f41d1-d471-47a7-9f9b-00ecfc28ea39" (UID: "560f41d1-d471-47a7-9f9b-00ecfc28ea39"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:52:33.340462 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.340422    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560f41d1-d471-47a7-9f9b-00ecfc28ea39-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "560f41d1-d471-47a7-9f9b-00ecfc28ea39" (UID: "560f41d1-d471-47a7-9f9b-00ecfc28ea39"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:52:33.340462 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.340452    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-api-access-wlfwm" (OuterVolumeSpecName: "kube-api-access-wlfwm") pod "560f41d1-d471-47a7-9f9b-00ecfc28ea39" (UID: "560f41d1-d471-47a7-9f9b-00ecfc28ea39"). InnerVolumeSpecName "kube-api-access-wlfwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:52:33.439176 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.439140    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:52:33.439176 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.439169    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/560f41d1-d471-47a7-9f9b-00ecfc28ea39-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:52:33.439176 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.439181    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlfwm\" (UniqueName: \"kubernetes.io/projected/560f41d1-d471-47a7-9f9b-00ecfc28ea39-kube-api-access-wlfwm\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:52:33.439405 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.439192    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/560f41d1-d471-47a7-9f9b-00ecfc28ea39-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:52:33.568641 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.568609    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"]
Sep 05 02:52:33.571985 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:33.571956    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tl5ks"]
Sep 05 02:52:35.544746 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:35.544714    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" path="/var/lib/kubelet/pods/560f41d1-d471-47a7-9f9b-00ecfc28ea39/volumes"
Sep 05 02:52:59.619040 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:59.619011    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:52:59.619963 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:52:59.619940    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:54:08.440453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:08.440410    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"]
Sep 05 02:54:08.441004 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:08.440820    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container" containerID="cri-o://afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3" gracePeriod=30
Sep 05 02:54:08.441004 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:08.440836    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kube-rbac-proxy" containerID="cri-o://6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae" gracePeriod=30
Sep 05 02:54:08.672323 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:08.672290    2555 generic.go:356] "Generic (PLEG): container finished" podID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerID="6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae" exitCode=2
Sep 05 02:54:08.672499 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:08.672371    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" event={"ID":"33e9b29a-e3b3-4efd-8ca9-365972d9b38e","Type":"ContainerDied","Data":"6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae"}
Sep 05 02:54:09.555736 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.555699    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"]
Sep 05 02:54:09.556097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556008    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kube-rbac-proxy"
Sep 05 02:54:09.556097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556020    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kube-rbac-proxy"
Sep 05 02:54:09.556097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556034    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kube-rbac-proxy"
Sep 05 02:54:09.556097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556040    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kube-rbac-proxy"
Sep 05 02:54:09.556097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556051    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kserve-container"
Sep 05 02:54:09.556097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556056    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kserve-container"
Sep 05 02:54:09.556097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556067    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="storage-initializer"
Sep 05 02:54:09.556097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556073    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="storage-initializer"
Sep 05 02:54:09.556097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556087    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="storage-initializer"
Sep 05 02:54:09.556097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556092    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="storage-initializer"
Sep 05 02:54:09.556097 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556100    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container"
Sep 05 02:54:09.556439 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556106    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container"
Sep 05 02:54:09.556439 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556157    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kserve-container"
Sep 05 02:54:09.556439 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556167    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kube-rbac-proxy"
Sep 05 02:54:09.556439 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556173    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="560f41d1-d471-47a7-9f9b-00ecfc28ea39" containerName="kube-rbac-proxy"
Sep 05 02:54:09.556439 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.556179    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc4a057f-325a-451f-8da9-a8eaff25a35b" containerName="kserve-container"
Sep 05 02:54:09.559340 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.559324    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.564223 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.564200    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 05 02:54:09.564368 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.564210    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 05 02:54:09.580280 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.580255    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"]
Sep 05 02:54:09.655630 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.655599    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.655630 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.655634    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnwg\" (UniqueName: \"kubernetes.io/projected/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-api-access-wrnwg\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.655843 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.655663    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.655843 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.655754    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21d5d55-bdd2-4753-b682-8a8566d5cae1-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.655843 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.655798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.757145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.757103    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.757316 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.757187    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.757316 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.757219    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnwg\" (UniqueName: \"kubernetes.io/projected/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-api-access-wrnwg\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.757316 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.757257    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.757316 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.757306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21d5d55-bdd2-4753-b682-8a8566d5cae1-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.757710 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.757688    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.757918 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.757899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.757957 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.757939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.759796 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.759775    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21d5d55-bdd2-4753-b682-8a8566d5cae1-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.767575 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.767555    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnwg\" (UniqueName: \"kubernetes.io/projected/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-api-access-wrnwg\") pod \"isvc-xgboost-predictor-7566d65bf7-hbfw4\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:09.810833 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.810753    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 05 02:54:09.816580 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.816520    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 05 02:54:09.868688 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:09.868646    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:10.162093 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:10.162068    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"]
Sep 05 02:54:10.163922 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:54:10.163891    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21d5d55_bdd2_4753_b682_8a8566d5cae1.slice/crio-8316a98a83e2a3a620dcb50d8e0fd339823aa5f50a6128922bd23e1dfe2221be WatchSource:0}: Error finding container 8316a98a83e2a3a620dcb50d8e0fd339823aa5f50a6128922bd23e1dfe2221be: Status 404 returned error can't find the container with id 8316a98a83e2a3a620dcb50d8e0fd339823aa5f50a6128922bd23e1dfe2221be
Sep 05 02:54:10.683828 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:10.683788    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" event={"ID":"c21d5d55-bdd2-4753-b682-8a8566d5cae1","Type":"ContainerStarted","Data":"8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184"}
Sep 05 02:54:10.684200 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:10.683862    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" event={"ID":"c21d5d55-bdd2-4753-b682-8a8566d5cae1","Type":"ContainerStarted","Data":"8316a98a83e2a3a620dcb50d8e0fd339823aa5f50a6128922bd23e1dfe2221be"}
Sep 05 02:54:13.086850 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.086827    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:54:13.191179 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.191092    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-proxy-tls\") pod \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") "
Sep 05 02:54:13.191179 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.191130    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whs57\" (UniqueName: \"kubernetes.io/projected/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-api-access-whs57\") pod \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") "
Sep 05 02:54:13.191179 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.191172    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-rbac-proxy-sar-config\") pod \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") "
Sep 05 02:54:13.191411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.191259    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kserve-provision-location\") pod \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") "
Sep 05 02:54:13.191411 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.191319    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-cabundle-cert\") pod \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\" (UID: \"33e9b29a-e3b3-4efd-8ca9-365972d9b38e\") "
Sep 05 02:54:13.191666 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.191632    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33e9b29a-e3b3-4efd-8ca9-365972d9b38e" (UID: "33e9b29a-e3b3-4efd-8ca9-365972d9b38e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:13.191787 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.191658    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "33e9b29a-e3b3-4efd-8ca9-365972d9b38e" (UID: "33e9b29a-e3b3-4efd-8ca9-365972d9b38e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:54:13.191787 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.191768    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "33e9b29a-e3b3-4efd-8ca9-365972d9b38e" (UID: "33e9b29a-e3b3-4efd-8ca9-365972d9b38e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:54:13.193842 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.193814    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-api-access-whs57" (OuterVolumeSpecName: "kube-api-access-whs57") pod "33e9b29a-e3b3-4efd-8ca9-365972d9b38e" (UID: "33e9b29a-e3b3-4efd-8ca9-365972d9b38e"). InnerVolumeSpecName "kube-api-access-whs57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:54:13.193842 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.193822    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33e9b29a-e3b3-4efd-8ca9-365972d9b38e" (UID: "33e9b29a-e3b3-4efd-8ca9-365972d9b38e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:54:13.292303 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.292270    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:54:13.292303 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.292299    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:54:13.292303 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.292308    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whs57\" (UniqueName: \"kubernetes.io/projected/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-api-access-whs57\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:54:13.292563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.292319    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:54:13.292563 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.292329    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e9b29a-e3b3-4efd-8ca9-365972d9b38e-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:54:13.554975 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.554945    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"]
Sep 05 02:54:13.555278 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.555266    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container"
Sep 05 02:54:13.555320 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.555280    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container"
Sep 05 02:54:13.555320 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.555295    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="storage-initializer"
Sep 05 02:54:13.555320 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.555300    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="storage-initializer"
Sep 05 02:54:13.555320 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.555317    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kube-rbac-proxy"
Sep 05 02:54:13.555443 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.555323    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kube-rbac-proxy"
Sep 05 02:54:13.555443 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.555373    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kserve-container"
Sep 05 02:54:13.555443 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.555381    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerName="kube-rbac-proxy"
Sep 05 02:54:13.558755 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.558739    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.561114 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.561094    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 05 02:54:13.561217 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.561198    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 05 02:54:13.570028 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.570006    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"]
Sep 05 02:54:13.695138 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.695108    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d05b20fb-178c-4749-96dc-e403d40abff9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.695302 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.695152    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.695302 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.695180    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmwr\" (UniqueName: \"kubernetes.io/projected/d05b20fb-178c-4749-96dc-e403d40abff9-kube-api-access-xnmwr\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.695302 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.695216    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.695302 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.695290    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05b20fb-178c-4749-96dc-e403d40abff9-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.699862 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.699837    2555 generic.go:356] "Generic (PLEG): container finished" podID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" containerID="afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3" exitCode=0
Sep 05 02:54:13.699966 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.699882    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" event={"ID":"33e9b29a-e3b3-4efd-8ca9-365972d9b38e","Type":"ContainerDied","Data":"afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3"}
Sep 05 02:54:13.699966 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.699914    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"
Sep 05 02:54:13.699966 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.699927    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2" event={"ID":"33e9b29a-e3b3-4efd-8ca9-365972d9b38e","Type":"ContainerDied","Data":"00f3fbf891535b5934c6c28b7a16c8aef34dac4bdb5b467aad6e0147992b49a1"}
Sep 05 02:54:13.699966 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.699953    2555 scope.go:117] "RemoveContainer" containerID="6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae"
Sep 05 02:54:13.708471 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.708455    2555 scope.go:117] "RemoveContainer" containerID="afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3"
Sep 05 02:54:13.715767 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.715751    2555 scope.go:117] "RemoveContainer" containerID="2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161"
Sep 05 02:54:13.718547 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.718505    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"]
Sep 05 02:54:13.723253 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.723233    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-42sq2"]
Sep 05 02:54:13.723875 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.723863    2555 scope.go:117] "RemoveContainer" containerID="6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae"
Sep 05 02:54:13.724124 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:54:13.724108    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae\": container with ID starting with 6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae not found: ID does not exist" containerID="6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae"
Sep 05 02:54:13.724177 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.724130    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae"} err="failed to get container status \"6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae\": rpc error: code = NotFound desc = could not find container \"6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae\": container with ID starting with 6717665246b7af22b23b2fc3e097c93d4d1d31e94c8063440712e697451763ae not found: ID does not exist"
Sep 05 02:54:13.724177 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.724147    2555 scope.go:117] "RemoveContainer" containerID="afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3"
Sep 05 02:54:13.724387 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:54:13.724371    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3\": container with ID starting with afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3 not found: ID does not exist" containerID="afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3"
Sep 05 02:54:13.724430 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.724392    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3"} err="failed to get container status \"afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3\": rpc error: code = NotFound desc = could not find container \"afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3\": container with ID starting with afe5f48b4ffb0de7132ad019bead9b5768f73e3bae7fe488d5e2ff86943b3dc3 not found: ID does not exist"
Sep 05 02:54:13.724430 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.724405    2555 scope.go:117] "RemoveContainer" containerID="2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161"
Sep 05 02:54:13.724643 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:54:13.724622    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161\": container with ID starting with 2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161 not found: ID does not exist" containerID="2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161"
Sep 05 02:54:13.724691 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.724646    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161"} err="failed to get container status \"2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161\": rpc error: code = NotFound desc = could not find container \"2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161\": container with ID starting with 2835b34c67e9c87e7495330f6c5ca6fcce019e55701949013fd5b82de6dfe161 not found: ID does not exist"
Sep 05 02:54:13.796606 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.796576    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05b20fb-178c-4749-96dc-e403d40abff9-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.796721 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.796646    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d05b20fb-178c-4749-96dc-e403d40abff9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.796772 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.796736    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.796772 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.796752    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmwr\" (UniqueName: \"kubernetes.io/projected/d05b20fb-178c-4749-96dc-e403d40abff9-kube-api-access-xnmwr\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.796876 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.796777    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.797035 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.797011    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d05b20fb-178c-4749-96dc-e403d40abff9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.797356 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.797333    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.797434 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.797401    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.798952 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.798932    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05b20fb-178c-4749-96dc-e403d40abff9-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.805658 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.805615    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmwr\" (UniqueName: \"kubernetes.io/projected/d05b20fb-178c-4749-96dc-e403d40abff9-kube-api-access-xnmwr\") pod \"isvc-canary-stop-predictor-79f67b4d78-vgmhb\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:13.869579 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:13.869529    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:14.164973 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:14.164939    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"]
Sep 05 02:54:14.167946 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:54:14.167918    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05b20fb_178c_4749_96dc_e403d40abff9.slice/crio-80b6280f636c6e88a51f69cbc181c2310ab15ef6cd356963c5e720e0db20aa9e WatchSource:0}: Error finding container 80b6280f636c6e88a51f69cbc181c2310ab15ef6cd356963c5e720e0db20aa9e: Status 404 returned error can't find the container with id 80b6280f636c6e88a51f69cbc181c2310ab15ef6cd356963c5e720e0db20aa9e
Sep 05 02:54:14.707708 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:14.707659    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" event={"ID":"d05b20fb-178c-4749-96dc-e403d40abff9","Type":"ContainerStarted","Data":"2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89"}
Sep 05 02:54:14.707909 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:14.707741    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" event={"ID":"d05b20fb-178c-4749-96dc-e403d40abff9","Type":"ContainerStarted","Data":"80b6280f636c6e88a51f69cbc181c2310ab15ef6cd356963c5e720e0db20aa9e"}
Sep 05 02:54:14.710268 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:14.710244    2555 generic.go:356] "Generic (PLEG): container finished" podID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerID="8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184" exitCode=0
Sep 05 02:54:14.710381 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:14.710317    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" event={"ID":"c21d5d55-bdd2-4753-b682-8a8566d5cae1","Type":"ContainerDied","Data":"8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184"}
Sep 05 02:54:14.711633 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:14.711615    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:54:15.544807 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:15.544771    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e9b29a-e3b3-4efd-8ca9-365972d9b38e" path="/var/lib/kubelet/pods/33e9b29a-e3b3-4efd-8ca9-365972d9b38e/volumes"
Sep 05 02:54:18.733808 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:18.733768    2555 generic.go:356] "Generic (PLEG): container finished" podID="d05b20fb-178c-4749-96dc-e403d40abff9" containerID="2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89" exitCode=0
Sep 05 02:54:18.734261 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:18.733888    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" event={"ID":"d05b20fb-178c-4749-96dc-e403d40abff9","Type":"ContainerDied","Data":"2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89"}
Sep 05 02:54:19.743619 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:19.743582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" event={"ID":"d05b20fb-178c-4749-96dc-e403d40abff9","Type":"ContainerStarted","Data":"6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6"}
Sep 05 02:54:19.744037 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:19.743647    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" event={"ID":"d05b20fb-178c-4749-96dc-e403d40abff9","Type":"ContainerStarted","Data":"278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6"}
Sep 05 02:54:19.744037 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:19.743869    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:19.764134 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:19.764074    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podStartSLOduration=3.030953021 podStartE2EDuration="6.764050155s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.733097134s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:54:19.760898856 +0000 UTC m=+2180.769395014" watchObservedRunningTime="2026-09-05 02:54:19.764050155 +0000 UTC m=+2180.772546295"
Sep 05 02:54:20.748819 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:20.748782    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:20.750461 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:20.750429    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 05 02:54:21.754118 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:21.754073    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 05 02:54:26.760996 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:26.760963    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:54:26.761884 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:26.761519    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 05 02:54:35.828830 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:35.828791    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" event={"ID":"c21d5d55-bdd2-4753-b682-8a8566d5cae1","Type":"ContainerStarted","Data":"08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a"}
Sep 05 02:54:35.829207 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:35.828856    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" event={"ID":"c21d5d55-bdd2-4753-b682-8a8566d5cae1","Type":"ContainerStarted","Data":"29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5"}
Sep 05 02:54:35.829207 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:35.829037    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:35.847505 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:35.847460    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podStartSLOduration=2.927704972 podStartE2EDuration="26.847445352s" totalImagesPullingTime="20.09187146s" totalInitContainerRuntime="3.82786892s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:54:35.845891667 +0000 UTC m=+2196.854387827" watchObservedRunningTime="2026-09-05 02:54:35.847445352 +0000 UTC m=+2196.855941492"
Sep 05 02:54:36.762224 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:36.762187    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 05 02:54:36.834943 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:36.834899    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:36.836162 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:36.836122    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 05 02:54:37.840800 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:37.840757    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 05 02:54:42.846159 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:42.846127    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:54:42.846654 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:42.846623    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 05 02:54:46.761582 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:46.761516    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 05 02:54:52.847154 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:52.847103    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 05 02:54:56.762118 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:54:56.762069    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 05 02:55:02.846811 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:02.846770    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 05 02:55:06.762342 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:06.762293    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 05 02:55:12.846751 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:12.846710    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 05 02:55:16.762313 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:16.762266    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 05 02:55:22.847007 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:22.846911    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 05 02:55:26.762413 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:26.762370    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 05 02:55:32.847399 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:32.847355    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 05 02:55:36.266227 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:36.266190    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"]
Sep 05 02:55:36.266683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:36.266453    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container" containerID="cri-o://278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6" gracePeriod=30
Sep 05 02:55:36.266683 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:36.266574    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kube-rbac-proxy" containerID="cri-o://6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6" gracePeriod=30
Sep 05 02:55:36.267748 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:36.267716    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:55:36.755110 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:36.755066    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 05 02:55:37.108946 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:37.108854    2555 generic.go:356] "Generic (PLEG): container finished" podID="d05b20fb-178c-4749-96dc-e403d40abff9" containerID="6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6" exitCode=2
Sep 05 02:55:37.109081 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:37.108934    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" event={"ID":"d05b20fb-178c-4749-96dc-e403d40abff9","Type":"ContainerDied","Data":"6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6"}
Sep 05 02:55:41.107300 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.107277    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:55:41.127694 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.127658    2555 generic.go:356] "Generic (PLEG): container finished" podID="d05b20fb-178c-4749-96dc-e403d40abff9" containerID="278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6" exitCode=0
Sep 05 02:55:41.127880 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.127732    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"
Sep 05 02:55:41.127880 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.127734    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" event={"ID":"d05b20fb-178c-4749-96dc-e403d40abff9","Type":"ContainerDied","Data":"278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6"}
Sep 05 02:55:41.127880 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.127851    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb" event={"ID":"d05b20fb-178c-4749-96dc-e403d40abff9","Type":"ContainerDied","Data":"80b6280f636c6e88a51f69cbc181c2310ab15ef6cd356963c5e720e0db20aa9e"}
Sep 05 02:55:41.128033 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.127885    2555 scope.go:117] "RemoveContainer" containerID="6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6"
Sep 05 02:55:41.129876 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.129852    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-kube-rbac-proxy-sar-config\") pod \"d05b20fb-178c-4749-96dc-e403d40abff9\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") "
Sep 05 02:55:41.129997 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.129898    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d05b20fb-178c-4749-96dc-e403d40abff9-kserve-provision-location\") pod \"d05b20fb-178c-4749-96dc-e403d40abff9\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") "
Sep 05 02:55:41.130049 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.130012    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-cabundle-cert\") pod \"d05b20fb-178c-4749-96dc-e403d40abff9\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") "
Sep 05 02:55:41.130139 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.130080    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05b20fb-178c-4749-96dc-e403d40abff9-proxy-tls\") pod \"d05b20fb-178c-4749-96dc-e403d40abff9\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") "
Sep 05 02:55:41.130139 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.130118    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnmwr\" (UniqueName: \"kubernetes.io/projected/d05b20fb-178c-4749-96dc-e403d40abff9-kube-api-access-xnmwr\") pod \"d05b20fb-178c-4749-96dc-e403d40abff9\" (UID: \"d05b20fb-178c-4749-96dc-e403d40abff9\") "
Sep 05 02:55:41.130475 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.130376    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05b20fb-178c-4749-96dc-e403d40abff9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d05b20fb-178c-4749-96dc-e403d40abff9" (UID: "d05b20fb-178c-4749-96dc-e403d40abff9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:41.130619 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.130577    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d05b20fb-178c-4749-96dc-e403d40abff9" (UID: "d05b20fb-178c-4749-96dc-e403d40abff9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:55:41.130619 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.130590    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d05b20fb-178c-4749-96dc-e403d40abff9" (UID: "d05b20fb-178c-4749-96dc-e403d40abff9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:55:41.132805 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.132780    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05b20fb-178c-4749-96dc-e403d40abff9-kube-api-access-xnmwr" (OuterVolumeSpecName: "kube-api-access-xnmwr") pod "d05b20fb-178c-4749-96dc-e403d40abff9" (UID: "d05b20fb-178c-4749-96dc-e403d40abff9"). InnerVolumeSpecName "kube-api-access-xnmwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:55:41.132906 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.132854    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05b20fb-178c-4749-96dc-e403d40abff9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d05b20fb-178c-4749-96dc-e403d40abff9" (UID: "d05b20fb-178c-4749-96dc-e403d40abff9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:55:41.148674 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.148644    2555 scope.go:117] "RemoveContainer" containerID="278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6"
Sep 05 02:55:41.156645 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.156621    2555 scope.go:117] "RemoveContainer" containerID="2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89"
Sep 05 02:55:41.163971 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.163952    2555 scope.go:117] "RemoveContainer" containerID="6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6"
Sep 05 02:55:41.164299 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:55:41.164278    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6\": container with ID starting with 6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6 not found: ID does not exist" containerID="6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6"
Sep 05 02:55:41.164362 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.164310    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6"} err="failed to get container status \"6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6\": rpc error: code = NotFound desc = could not find container \"6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6\": container with ID starting with 6340f2224bd7eb24e96003ad84bc2133475f4d3c430c277353ecb71549d59ba6 not found: ID does not exist"
Sep 05 02:55:41.164362 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.164330    2555 scope.go:117] "RemoveContainer" containerID="278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6"
Sep 05 02:55:41.164605 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:55:41.164584    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6\": container with ID starting with 278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6 not found: ID does not exist" containerID="278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6"
Sep 05 02:55:41.164680 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.164616    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6"} err="failed to get container status \"278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6\": rpc error: code = NotFound desc = could not find container \"278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6\": container with ID starting with 278b51e745bfbb6ee740990b7553684cd8ad8130e884a8d19db0e436a6d6e1d6 not found: ID does not exist"
Sep 05 02:55:41.164680 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.164638    2555 scope.go:117] "RemoveContainer" containerID="2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89"
Sep 05 02:55:41.164917 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:55:41.164899    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89\": container with ID starting with 2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89 not found: ID does not exist" containerID="2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89"
Sep 05 02:55:41.164961 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.164925    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89"} err="failed to get container status \"2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89\": rpc error: code = NotFound desc = could not find container \"2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89\": container with ID starting with 2ea7155157b59466b63b78fa2bc444885f091b2b2a053cef9cebe220f7dc5c89 not found: ID does not exist"
Sep 05 02:55:41.231079 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.230984    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d05b20fb-178c-4749-96dc-e403d40abff9-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:55:41.231079 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.231016    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmwr\" (UniqueName: \"kubernetes.io/projected/d05b20fb-178c-4749-96dc-e403d40abff9-kube-api-access-xnmwr\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:55:41.231079 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.231026    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:55:41.231079 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.231037    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d05b20fb-178c-4749-96dc-e403d40abff9-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:55:41.231079 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.231046    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d05b20fb-178c-4749-96dc-e403d40abff9-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:55:41.451684 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.451641    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"]
Sep 05 02:55:41.455378 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.455351    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-vgmhb"]
Sep 05 02:55:41.544132 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:41.544052    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" path="/var/lib/kubelet/pods/d05b20fb-178c-4749-96dc-e403d40abff9/volumes"
Sep 05 02:55:42.847652 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:42.847618    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:55:49.439658 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:49.439621    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"]
Sep 05 02:55:49.440078 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:49.439849    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container" containerID="cri-o://29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5" gracePeriod=30
Sep 05 02:55:49.440078 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:49.439940    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kube-rbac-proxy" containerID="cri-o://08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a" gracePeriod=30
Sep 05 02:55:50.170699 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:50.170658    2555 generic.go:356] "Generic (PLEG): container finished" podID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerID="08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a" exitCode=2
Sep 05 02:55:50.170886 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:50.170714    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" event={"ID":"c21d5d55-bdd2-4753-b682-8a8566d5cae1","Type":"ContainerDied","Data":"08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a"}
Sep 05 02:55:52.841675 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:52.841636    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 05 02:55:52.847021 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:52.846990    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 05 02:55:52.990065 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:52.990040    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:55:53.038138 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.038109    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kserve-provision-location\") pod \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") "
Sep 05 02:55:53.038334 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.038153    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-rbac-proxy-sar-config\") pod \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") "
Sep 05 02:55:53.038334 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.038196    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrnwg\" (UniqueName: \"kubernetes.io/projected/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-api-access-wrnwg\") pod \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") "
Sep 05 02:55:53.038471 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.038327    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21d5d55-bdd2-4753-b682-8a8566d5cae1-proxy-tls\") pod \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") "
Sep 05 02:55:53.038471 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.038408    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-cabundle-cert\") pod \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\" (UID: \"c21d5d55-bdd2-4753-b682-8a8566d5cae1\") "
Sep 05 02:55:53.038607 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.038502    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c21d5d55-bdd2-4753-b682-8a8566d5cae1" (UID: "c21d5d55-bdd2-4753-b682-8a8566d5cae1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.038796 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.038743    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c21d5d55-bdd2-4753-b682-8a8566d5cae1" (UID: "c21d5d55-bdd2-4753-b682-8a8566d5cae1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:55:53.038890 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.038749    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c21d5d55-bdd2-4753-b682-8a8566d5cae1" (UID: "c21d5d55-bdd2-4753-b682-8a8566d5cae1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:55:53.038890 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.038782    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.040906 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.040878    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21d5d55-bdd2-4753-b682-8a8566d5cae1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c21d5d55-bdd2-4753-b682-8a8566d5cae1" (UID: "c21d5d55-bdd2-4753-b682-8a8566d5cae1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:55:53.041040 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.040888    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-api-access-wrnwg" (OuterVolumeSpecName: "kube-api-access-wrnwg") pod "c21d5d55-bdd2-4753-b682-8a8566d5cae1" (UID: "c21d5d55-bdd2-4753-b682-8a8566d5cae1"). InnerVolumeSpecName "kube-api-access-wrnwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:55:53.139556 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.139502    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.139726 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.139567    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrnwg\" (UniqueName: \"kubernetes.io/projected/c21d5d55-bdd2-4753-b682-8a8566d5cae1-kube-api-access-wrnwg\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.139726 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.139581    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21d5d55-bdd2-4753-b682-8a8566d5cae1-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.139726 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.139590    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c21d5d55-bdd2-4753-b682-8a8566d5cae1-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.186074 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.186037    2555 generic.go:356] "Generic (PLEG): container finished" podID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerID="29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5" exitCode=0
Sep 05 02:55:53.186244 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.186089    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"
Sep 05 02:55:53.186244 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.186105    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" event={"ID":"c21d5d55-bdd2-4753-b682-8a8566d5cae1","Type":"ContainerDied","Data":"29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5"}
Sep 05 02:55:53.186244 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.186166    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4" event={"ID":"c21d5d55-bdd2-4753-b682-8a8566d5cae1","Type":"ContainerDied","Data":"8316a98a83e2a3a620dcb50d8e0fd339823aa5f50a6128922bd23e1dfe2221be"}
Sep 05 02:55:53.186244 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.186195    2555 scope.go:117] "RemoveContainer" containerID="08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a"
Sep 05 02:55:53.196448 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.196426    2555 scope.go:117] "RemoveContainer" containerID="29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5"
Sep 05 02:55:53.203680 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.203657    2555 scope.go:117] "RemoveContainer" containerID="8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184"
Sep 05 02:55:53.210762 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.210743    2555 scope.go:117] "RemoveContainer" containerID="08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a"
Sep 05 02:55:53.211082 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:55:53.211058    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a\": container with ID starting with 08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a not found: ID does not exist" containerID="08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a"
Sep 05 02:55:53.211178 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.211090    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a"} err="failed to get container status \"08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a\": rpc error: code = NotFound desc = could not find container \"08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a\": container with ID starting with 08986162fc9eaef47cd3caceefb6b3a2ad39c518b9bb0c6fbc35eeadd3959f1a not found: ID does not exist"
Sep 05 02:55:53.211178 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.211108    2555 scope.go:117] "RemoveContainer" containerID="29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5"
Sep 05 02:55:53.211385 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:55:53.211361    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5\": container with ID starting with 29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5 not found: ID does not exist" containerID="29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5"
Sep 05 02:55:53.211426 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.211396    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5"} err="failed to get container status \"29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5\": rpc error: code = NotFound desc = could not find container \"29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5\": container with ID starting with 29350a427462ef8aac878e6833eb40af96bff0090b1c2792b3833297439754f5 not found: ID does not exist"
Sep 05 02:55:53.211426 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.211414    2555 scope.go:117] "RemoveContainer" containerID="8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184"
Sep 05 02:55:53.211775 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:55:53.211748    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184\": container with ID starting with 8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184 not found: ID does not exist" containerID="8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184"
Sep 05 02:55:53.211898 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.211780    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184"} err="failed to get container status \"8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184\": rpc error: code = NotFound desc = could not find container \"8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184\": container with ID starting with 8972058a40df4c506cdd2bf8415d08283e6cd3220362325494d7ddaaaa8bd184 not found: ID does not exist"
Sep 05 02:55:53.214610 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.214588    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"]
Sep 05 02:55:53.218942 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.218921    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-hbfw4"]
Sep 05 02:55:53.545604 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.545486    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" path="/var/lib/kubelet/pods/c21d5d55-bdd2-4753-b682-8a8566d5cae1/volumes"
Sep 05 02:55:53.870108 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870026    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"]
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870351    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kube-rbac-proxy"
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870362    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kube-rbac-proxy"
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870372    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kube-rbac-proxy"
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870377    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kube-rbac-proxy"
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870394    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="storage-initializer"
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870399    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="storage-initializer"
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870405    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container"
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870410    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container"
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870416    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container"
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870422    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container"
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870434    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="storage-initializer"
Sep 05 02:55:53.870453 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870440    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="storage-initializer"
Sep 05 02:55:53.870862 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870488    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kube-rbac-proxy"
Sep 05 02:55:53.870862 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870498    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kube-rbac-proxy"
Sep 05 02:55:53.870862 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870504    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d05b20fb-178c-4749-96dc-e403d40abff9" containerName="kserve-container"
Sep 05 02:55:53.870862 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.870512    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c21d5d55-bdd2-4753-b682-8a8566d5cae1" containerName="kserve-container"
Sep 05 02:55:53.875132 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.875113    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:53.877879 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.877858    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:55:53.888178 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.888159    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 05 02:55:53.888290 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.888188    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:55:53.888393 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.888381    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:55:53.888431 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.888394    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:55:53.888431 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.888410    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 05 02:55:53.888431 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.888426    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:55:53.898512 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.898484    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"]
Sep 05 02:55:53.943962 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.943930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cgqq\" (UniqueName: \"kubernetes.io/projected/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-api-access-9cgqq\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:53.944118 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.943979    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86e8777d-93c0-4ab0-af75-eb81789e6b9d-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:53.944118 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.943999    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:53.944118 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.944051    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:53.944262 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:53.944121    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.044870 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.044833    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.045057 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.044894    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.045057 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.044928    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cgqq\" (UniqueName: \"kubernetes.io/projected/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-api-access-9cgqq\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.045057 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.044963    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86e8777d-93c0-4ab0-af75-eb81789e6b9d-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.045057 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.044980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.045426 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.045403    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.045679 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.045660    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.045738 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.045704    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.047526 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.047504    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86e8777d-93c0-4ab0-af75-eb81789e6b9d-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.053660 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.053632    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cgqq\" (UniqueName: \"kubernetes.io/projected/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-api-access-9cgqq\") pod \"isvc-lightgbm-predictor-54f85ddf88-tttqc\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.184873 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.184831    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:55:54.460928 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:54.460901    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"]
Sep 05 02:55:54.462382 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:55:54.462351    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e8777d_93c0_4ab0_af75_eb81789e6b9d.slice/crio-39d415e283d212d4041ad30fbae97a0b9fd948b7e93da426055ef214cf4d9202 WatchSource:0}: Error finding container 39d415e283d212d4041ad30fbae97a0b9fd948b7e93da426055ef214cf4d9202: Status 404 returned error can't find the container with id 39d415e283d212d4041ad30fbae97a0b9fd948b7e93da426055ef214cf4d9202
Sep 05 02:55:55.202146 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:55.202108    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" event={"ID":"86e8777d-93c0-4ab0-af75-eb81789e6b9d","Type":"ContainerStarted","Data":"65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b"}
Sep 05 02:55:55.202518 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:55.202197    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" event={"ID":"86e8777d-93c0-4ab0-af75-eb81789e6b9d","Type":"ContainerStarted","Data":"39d415e283d212d4041ad30fbae97a0b9fd948b7e93da426055ef214cf4d9202"}
Sep 05 02:55:59.221175 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:59.221145    2555 generic.go:356] "Generic (PLEG): container finished" podID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerID="65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b" exitCode=0
Sep 05 02:55:59.221564 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:55:59.221196    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" event={"ID":"86e8777d-93c0-4ab0-af75-eb81789e6b9d","Type":"ContainerDied","Data":"65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b"}
Sep 05 02:56:06.259145 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:06.259112    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" event={"ID":"86e8777d-93c0-4ab0-af75-eb81789e6b9d","Type":"ContainerStarted","Data":"4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192"}
Sep 05 02:56:06.259529 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:06.259176    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" event={"ID":"86e8777d-93c0-4ab0-af75-eb81789e6b9d","Type":"ContainerStarted","Data":"f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7"}
Sep 05 02:56:06.259529 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:06.259365    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:56:06.279012 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:06.278961    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podStartSLOduration=2.579153614 podStartE2EDuration="13.278946612s" totalImagesPullingTime="6.441911421s" totalInitContainerRuntime="4.257881577s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:56:06.277737788 +0000 UTC m=+2287.286233963" watchObservedRunningTime="2026-09-05 02:56:06.278946612 +0000 UTC m=+2287.287442750"
Sep 05 02:56:07.263721 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:07.263690    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:56:07.265205 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:07.265174    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 05 02:56:08.267598 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:08.267556    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 05 02:56:13.272127 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:13.272096    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:56:13.272657 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:13.272631    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 05 02:56:23.273213 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:23.273166    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 05 02:56:33.273325 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:33.273282    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 05 02:56:43.272677 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:43.272633    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 05 02:56:53.273562 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:56:53.273453    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 05 02:57:03.272610 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:03.272570    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 05 02:57:13.273489 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:13.273450    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 05 02:57:23.273707 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:23.273676    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:57:23.889199 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:23.889166    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"]
Sep 05 02:57:23.889410 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:23.889369    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container" containerID="cri-o://f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7" gracePeriod=30
Sep 05 02:57:23.889507 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:23.889476    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kube-rbac-proxy" containerID="cri-o://4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192" gracePeriod=30
Sep 05 02:57:24.603214 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:24.603179    2555 generic.go:356] "Generic (PLEG): container finished" podID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerID="4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192" exitCode=2
Sep 05 02:57:24.603688 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:24.603251    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" event={"ID":"86e8777d-93c0-4ab0-af75-eb81789e6b9d","Type":"ContainerDied","Data":"4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192"}
Sep 05 02:57:28.268670 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.268618    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 05 02:57:28.533881 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.533820    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:57:28.620878 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.620845    2555 generic.go:356] "Generic (PLEG): container finished" podID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerID="f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7" exitCode=0
Sep 05 02:57:28.621043 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.620904    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"
Sep 05 02:57:28.621043 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.620931    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" event={"ID":"86e8777d-93c0-4ab0-af75-eb81789e6b9d","Type":"ContainerDied","Data":"f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7"}
Sep 05 02:57:28.621043 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.620997    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc" event={"ID":"86e8777d-93c0-4ab0-af75-eb81789e6b9d","Type":"ContainerDied","Data":"39d415e283d212d4041ad30fbae97a0b9fd948b7e93da426055ef214cf4d9202"}
Sep 05 02:57:28.621043 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.621025    2555 scope.go:117] "RemoveContainer" containerID="4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192"
Sep 05 02:57:28.629275 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.629258    2555 scope.go:117] "RemoveContainer" containerID="f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7"
Sep 05 02:57:28.635381 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.635357    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cgqq\" (UniqueName: \"kubernetes.io/projected/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-api-access-9cgqq\") pod \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") "
Sep 05 02:57:28.635481 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.635407    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86e8777d-93c0-4ab0-af75-eb81789e6b9d-proxy-tls\") pod \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") "
Sep 05 02:57:28.635481 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.635463    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-cabundle-cert\") pod \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") "
Sep 05 02:57:28.635582 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.635514    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kserve-provision-location\") pod \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") "
Sep 05 02:57:28.635582 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.635573    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-rbac-proxy-sar-config\") pod \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\" (UID: \"86e8777d-93c0-4ab0-af75-eb81789e6b9d\") "
Sep 05 02:57:28.635981 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.635866    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86e8777d-93c0-4ab0-af75-eb81789e6b9d" (UID: "86e8777d-93c0-4ab0-af75-eb81789e6b9d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:57:28.635981 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.635947    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86e8777d-93c0-4ab0-af75-eb81789e6b9d" (UID: "86e8777d-93c0-4ab0-af75-eb81789e6b9d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:28.636156 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.636075    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86e8777d-93c0-4ab0-af75-eb81789e6b9d" (UID: "86e8777d-93c0-4ab0-af75-eb81789e6b9d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:57:28.636211 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.636198    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:57:28.636256 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.636223    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:57:28.636256 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.636241    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:57:28.636794 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.636777    2555 scope.go:117] "RemoveContainer" containerID="65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b"
Sep 05 02:57:28.637711 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.637688    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e8777d-93c0-4ab0-af75-eb81789e6b9d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86e8777d-93c0-4ab0-af75-eb81789e6b9d" (UID: "86e8777d-93c0-4ab0-af75-eb81789e6b9d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:57:28.637851 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.637809    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-api-access-9cgqq" (OuterVolumeSpecName: "kube-api-access-9cgqq") pod "86e8777d-93c0-4ab0-af75-eb81789e6b9d" (UID: "86e8777d-93c0-4ab0-af75-eb81789e6b9d"). InnerVolumeSpecName "kube-api-access-9cgqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:57:28.652311 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.652295    2555 scope.go:117] "RemoveContainer" containerID="4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192"
Sep 05 02:57:28.652596 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:57:28.652526    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192\": container with ID starting with 4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192 not found: ID does not exist" containerID="4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192"
Sep 05 02:57:28.652666 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.652610    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192"} err="failed to get container status \"4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192\": rpc error: code = NotFound desc = could not find container \"4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192\": container with ID starting with 4086f229ce59d5a6d79abe8d1160a96c211954c508bb8222a055f143e3a86192 not found: ID does not exist"
Sep 05 02:57:28.652666 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.652626    2555 scope.go:117] "RemoveContainer" containerID="f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7"
Sep 05 02:57:28.652892 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:57:28.652864    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7\": container with ID starting with f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7 not found: ID does not exist" containerID="f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7"
Sep 05 02:57:28.652929 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.652897    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7"} err="failed to get container status \"f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7\": rpc error: code = NotFound desc = could not find container \"f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7\": container with ID starting with f556ff25aeb240b9a04bd0e7d0ec4f34a07cf9e7c59f1ad03a95ab235b0ea7a7 not found: ID does not exist"
Sep 05 02:57:28.652929 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.652915    2555 scope.go:117] "RemoveContainer" containerID="65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b"
Sep 05 02:57:28.653130 ip-10-0-135-239 kubenswrapper[2555]: E0905 02:57:28.653113    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b\": container with ID starting with 65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b not found: ID does not exist" containerID="65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b"
Sep 05 02:57:28.653184 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.653134    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b"} err="failed to get container status \"65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b\": rpc error: code = NotFound desc = could not find container \"65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b\": container with ID starting with 65b012e15062a9fcb964203b8b4b921eccac1a8dbc38127d75c11c541a33053b not found: ID does not exist"
Sep 05 02:57:28.737079 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.737052    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cgqq\" (UniqueName: \"kubernetes.io/projected/86e8777d-93c0-4ab0-af75-eb81789e6b9d-kube-api-access-9cgqq\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:57:28.737079 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.737075    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86e8777d-93c0-4ab0-af75-eb81789e6b9d-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:57:28.950153 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.950116    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"]
Sep 05 02:57:28.951711 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:28.951684    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-tttqc"]
Sep 05 02:57:29.090299 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.090264    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"]
Sep 05 02:57:29.090657 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.090643    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kube-rbac-proxy"
Sep 05 02:57:29.090713 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.090659    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kube-rbac-proxy"
Sep 05 02:57:29.090713 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.090679    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="storage-initializer"
Sep 05 02:57:29.090713 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.090684    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="storage-initializer"
Sep 05 02:57:29.090713 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.090693    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container"
Sep 05 02:57:29.090713 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.090703    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container"
Sep 05 02:57:29.090867 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.090756    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kube-rbac-proxy"
Sep 05 02:57:29.090867 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.090766    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" containerName="kserve-container"
Sep 05 02:57:29.095091 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.095070    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.100126 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.100103    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:57:29.100253 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.100188    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:57:29.100328 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.100280    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:57:29.100328 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.100303    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 05 02:57:29.100440 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.100364    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:57:29.100440 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.100428    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 05 02:57:29.100814 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.100795    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:57:29.107269 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.107247    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"]
Sep 05 02:57:29.241027 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.240935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh6cb\" (UniqueName: \"kubernetes.io/projected/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-api-access-bh6cb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.241027 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.240998    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.241214 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.241033    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01a22f6c-8802-47bf-b40c-ab9d041775ff-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.241214 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.241058    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.241214 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.241084    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01a22f6c-8802-47bf-b40c-ab9d041775ff-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.342023 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.341990    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.342494 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.342035    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01a22f6c-8802-47bf-b40c-ab9d041775ff-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.342494 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.342170    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.342494 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.342238    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01a22f6c-8802-47bf-b40c-ab9d041775ff-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.342494 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.342296    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh6cb\" (UniqueName: \"kubernetes.io/projected/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-api-access-bh6cb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.342494 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.342389    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01a22f6c-8802-47bf-b40c-ab9d041775ff-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.342786 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.342768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.342825 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.342797    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.344790 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.344773    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01a22f6c-8802-47bf-b40c-ab9d041775ff-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.351331 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.351310    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh6cb\" (UniqueName: \"kubernetes.io/projected/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-api-access-bh6cb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.405333 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.405284    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:29.545255 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.545152    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86e8777d-93c0-4ab0-af75-eb81789e6b9d" path="/var/lib/kubelet/pods/86e8777d-93c0-4ab0-af75-eb81789e6b9d/volumes"
Sep 05 02:57:29.678840 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:29.678807    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"]
Sep 05 02:57:29.682116 ip-10-0-135-239 kubenswrapper[2555]: W0905 02:57:29.682072    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a22f6c_8802_47bf_b40c_ab9d041775ff.slice/crio-dd4520ec35cb4ee8e7b8134bed34947b7e6befe9471487f5584ddec42d6fa9ae WatchSource:0}: Error finding container dd4520ec35cb4ee8e7b8134bed34947b7e6befe9471487f5584ddec42d6fa9ae: Status 404 returned error can't find the container with id dd4520ec35cb4ee8e7b8134bed34947b7e6befe9471487f5584ddec42d6fa9ae
Sep 05 02:57:30.634697 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:30.634657    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" event={"ID":"01a22f6c-8802-47bf-b40c-ab9d041775ff","Type":"ContainerStarted","Data":"6b7d3fb3cba79c95a2aa3a6b3169bef584a17b706cfb6bc120e03e29d485cad2"}
Sep 05 02:57:30.635069 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:30.634711    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" event={"ID":"01a22f6c-8802-47bf-b40c-ab9d041775ff","Type":"ContainerStarted","Data":"dd4520ec35cb4ee8e7b8134bed34947b7e6befe9471487f5584ddec42d6fa9ae"}
Sep 05 02:57:33.648955 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:33.648920    2555 generic.go:356] "Generic (PLEG): container finished" podID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerID="6b7d3fb3cba79c95a2aa3a6b3169bef584a17b706cfb6bc120e03e29d485cad2" exitCode=0
Sep 05 02:57:33.649327 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:33.648967    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" event={"ID":"01a22f6c-8802-47bf-b40c-ab9d041775ff","Type":"ContainerDied","Data":"6b7d3fb3cba79c95a2aa3a6b3169bef584a17b706cfb6bc120e03e29d485cad2"}
Sep 05 02:57:34.655588 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:34.655553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" event={"ID":"01a22f6c-8802-47bf-b40c-ab9d041775ff","Type":"ContainerStarted","Data":"5b179f17eac5220506eb9e53954996ff58973572e97166cc50a18517aeecf3fc"}
Sep 05 02:57:34.656009 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:34.655612    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" event={"ID":"01a22f6c-8802-47bf-b40c-ab9d041775ff","Type":"ContainerStarted","Data":"39d8e0149afa0954f86a05f82f687277edee00cfd70bd23392b69f00c7776f1b"}
Sep 05 02:57:34.656009 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:34.655879    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:34.656130 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:34.656031    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:34.657473 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:34.657448    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 05 02:57:34.674050 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:34.673996    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podStartSLOduration=1.901292014 podStartE2EDuration="5.673980702s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.772688688s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:57:34.673422949 +0000 UTC m=+2375.681919088" watchObservedRunningTime="2026-09-05 02:57:34.673980702 +0000 UTC m=+2375.682476842"
Sep 05 02:57:35.660324 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:35.660278    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 05 02:57:40.665305 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:40.665276    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:57:40.665796 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:40.665767    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 05 02:57:50.666523 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:50.666477    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 05 02:57:59.649135 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:59.649103    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:57:59.650310 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:57:59.650289    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 02:58:00.666307 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:58:00.666268    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 05 02:58:10.666624 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:58:10.666584    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 05 02:58:20.666106 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:58:20.666068    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 05 02:58:30.666456 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:58:30.666367    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 05 02:58:40.666013 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:58:40.665965    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 05 02:58:50.666733 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:58:50.666698    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:58:59.340734 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:58:59.340694    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"]
Sep 05 02:58:59.341142 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:58:59.340970    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container" containerID="cri-o://39d8e0149afa0954f86a05f82f687277edee00cfd70bd23392b69f00c7776f1b" gracePeriod=30
Sep 05 02:58:59.341142 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:58:59.341040    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kube-rbac-proxy" containerID="cri-o://5b179f17eac5220506eb9e53954996ff58973572e97166cc50a18517aeecf3fc" gracePeriod=30
Sep 05 02:59:00.042315 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:00.042276    2555 generic.go:356] "Generic (PLEG): container finished" podID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerID="5b179f17eac5220506eb9e53954996ff58973572e97166cc50a18517aeecf3fc" exitCode=2
Sep 05 02:59:00.042484 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:00.042347    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" event={"ID":"01a22f6c-8802-47bf-b40c-ab9d041775ff","Type":"ContainerDied","Data":"5b179f17eac5220506eb9e53954996ff58973572e97166cc50a18517aeecf3fc"}
Sep 05 02:59:00.661291 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:00.661247    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 05 02:59:00.666070 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:00.666040    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 05 02:59:04.064324 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.064285    2555 generic.go:356] "Generic (PLEG): container finished" podID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerID="39d8e0149afa0954f86a05f82f687277edee00cfd70bd23392b69f00c7776f1b" exitCode=0
Sep 05 02:59:04.064674 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.064317    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" event={"ID":"01a22f6c-8802-47bf-b40c-ab9d041775ff","Type":"ContainerDied","Data":"39d8e0149afa0954f86a05f82f687277edee00cfd70bd23392b69f00c7776f1b"}
Sep 05 02:59:04.183622 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.183600    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:59:04.224785 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.224746    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01a22f6c-8802-47bf-b40c-ab9d041775ff-proxy-tls\") pod \"01a22f6c-8802-47bf-b40c-ab9d041775ff\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") "
Sep 05 02:59:04.224968 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.224797    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-rbac-proxy-sar-config\") pod \"01a22f6c-8802-47bf-b40c-ab9d041775ff\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") "
Sep 05 02:59:04.224968 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.224825    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01a22f6c-8802-47bf-b40c-ab9d041775ff-kserve-provision-location\") pod \"01a22f6c-8802-47bf-b40c-ab9d041775ff\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") "
Sep 05 02:59:04.224968 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.224870    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh6cb\" (UniqueName: \"kubernetes.io/projected/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-api-access-bh6cb\") pod \"01a22f6c-8802-47bf-b40c-ab9d041775ff\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") "
Sep 05 02:59:04.224968 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.224898    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-cabundle-cert\") pod \"01a22f6c-8802-47bf-b40c-ab9d041775ff\" (UID: \"01a22f6c-8802-47bf-b40c-ab9d041775ff\") "
Sep 05 02:59:04.225213 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.225136    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01a22f6c-8802-47bf-b40c-ab9d041775ff" (UID: "01a22f6c-8802-47bf-b40c-ab9d041775ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:59:04.225297 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.225271    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a22f6c-8802-47bf-b40c-ab9d041775ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01a22f6c-8802-47bf-b40c-ab9d041775ff" (UID: "01a22f6c-8802-47bf-b40c-ab9d041775ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:59:04.225366 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.225334    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01a22f6c-8802-47bf-b40c-ab9d041775ff" (UID: "01a22f6c-8802-47bf-b40c-ab9d041775ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:59:04.226986 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.226954    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-api-access-bh6cb" (OuterVolumeSpecName: "kube-api-access-bh6cb") pod "01a22f6c-8802-47bf-b40c-ab9d041775ff" (UID: "01a22f6c-8802-47bf-b40c-ab9d041775ff"). InnerVolumeSpecName "kube-api-access-bh6cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:59:04.227110 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.227000    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a22f6c-8802-47bf-b40c-ab9d041775ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01a22f6c-8802-47bf-b40c-ab9d041775ff" (UID: "01a22f6c-8802-47bf-b40c-ab9d041775ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:59:04.326115 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.326033    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01a22f6c-8802-47bf-b40c-ab9d041775ff-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:59:04.326115 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.326061    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:59:04.326115 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.326070    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01a22f6c-8802-47bf-b40c-ab9d041775ff-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:59:04.326115 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.326080    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh6cb\" (UniqueName: \"kubernetes.io/projected/01a22f6c-8802-47bf-b40c-ab9d041775ff-kube-api-access-bh6cb\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:59:04.326115 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:04.326090    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01a22f6c-8802-47bf-b40c-ab9d041775ff-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 02:59:05.070127 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:05.070091    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2" event={"ID":"01a22f6c-8802-47bf-b40c-ab9d041775ff","Type":"ContainerDied","Data":"dd4520ec35cb4ee8e7b8134bed34947b7e6befe9471487f5584ddec42d6fa9ae"}
Sep 05 02:59:05.070567 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:05.070159    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"
Sep 05 02:59:05.070567 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:05.070180    2555 scope.go:117] "RemoveContainer" containerID="5b179f17eac5220506eb9e53954996ff58973572e97166cc50a18517aeecf3fc"
Sep 05 02:59:05.078904 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:05.078889    2555 scope.go:117] "RemoveContainer" containerID="39d8e0149afa0954f86a05f82f687277edee00cfd70bd23392b69f00c7776f1b"
Sep 05 02:59:05.086163 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:05.086144    2555 scope.go:117] "RemoveContainer" containerID="6b7d3fb3cba79c95a2aa3a6b3169bef584a17b706cfb6bc120e03e29d485cad2"
Sep 05 02:59:05.092122 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:05.092096    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"]
Sep 05 02:59:05.095758 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:05.095732    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d92k2"]
Sep 05 02:59:05.543990 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:05.543959    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" path="/var/lib/kubelet/pods/01a22f6c-8802-47bf-b40c-ab9d041775ff/volumes"
Sep 05 02:59:59.793882 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.793768    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"]
Sep 05 02:59:59.797048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.794089    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="storage-initializer"
Sep 05 02:59:59.797048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.794101    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="storage-initializer"
Sep 05 02:59:59.797048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.794119    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kube-rbac-proxy"
Sep 05 02:59:59.797048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.794125    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kube-rbac-proxy"
Sep 05 02:59:59.797048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.794132    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container"
Sep 05 02:59:59.797048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.794138    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container"
Sep 05 02:59:59.797048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.794186    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kserve-container"
Sep 05 02:59:59.797048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.794193    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="01a22f6c-8802-47bf-b40c-ab9d041775ff" containerName="kube-rbac-proxy"
Sep 05 02:59:59.797048 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.797046    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 02:59:59.803656 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.803271    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:59:59.803810 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.803785    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:59:59.803810 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.803790    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 05 02:59:59.804062 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.803862    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 02:59:59.804062 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.803807    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:59:59.804062 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.803886    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 05 02:59:59.804428 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.804412    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:59:59.808490 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.808466    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"]
Sep 05 02:59:59.909385 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.909276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 02:59:59.909385 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.909322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 02:59:59.909477 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.909387    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81680a36-5835-4bb1-beb0-1cef04afefcf-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 02:59:59.909477 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.909435    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81680a36-5835-4bb1-beb0-1cef04afefcf-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 02:59:59.909586 ip-10-0-135-239 kubenswrapper[2555]: I0905 02:59:59.909554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-878rx\" (UniqueName: \"kubernetes.io/projected/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-api-access-878rx\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.010927 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.010886    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.011126 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.010944    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.011126 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.010967    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81680a36-5835-4bb1-beb0-1cef04afefcf-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.011126 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.010992    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81680a36-5835-4bb1-beb0-1cef04afefcf-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.011126 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.011054    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-878rx\" (UniqueName: \"kubernetes.io/projected/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-api-access-878rx\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.011474 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.011454    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81680a36-5835-4bb1-beb0-1cef04afefcf-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.011635 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.011612    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.011743 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.011722    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.013473 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.013453    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81680a36-5835-4bb1-beb0-1cef04afefcf-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.018466 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.018436    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-878rx\" (UniqueName: \"kubernetes.io/projected/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-api-access-878rx\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.107254 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.107162    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:00.387383 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:00.387336    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"]
Sep 05 03:00:00.389831 ip-10-0-135-239 kubenswrapper[2555]: W0905 03:00:00.389797    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81680a36_5835_4bb1_beb0_1cef04afefcf.slice/crio-8db93b7cd148eddb6a29a0e893553c98de25a740cf400d69fb4224e58af256f5 WatchSource:0}: Error finding container 8db93b7cd148eddb6a29a0e893553c98de25a740cf400d69fb4224e58af256f5: Status 404 returned error can't find the container with id 8db93b7cd148eddb6a29a0e893553c98de25a740cf400d69fb4224e58af256f5
Sep 05 03:00:01.322149 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:01.322114    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" event={"ID":"81680a36-5835-4bb1-beb0-1cef04afefcf","Type":"ContainerStarted","Data":"8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c"}
Sep 05 03:00:01.322547 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:01.322172    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" event={"ID":"81680a36-5835-4bb1-beb0-1cef04afefcf","Type":"ContainerStarted","Data":"8db93b7cd148eddb6a29a0e893553c98de25a740cf400d69fb4224e58af256f5"}
Sep 05 03:00:05.341324 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:05.341290    2555 generic.go:356] "Generic (PLEG): container finished" podID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerID="8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c" exitCode=0
Sep 05 03:00:05.341712 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:05.341373    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" event={"ID":"81680a36-5835-4bb1-beb0-1cef04afefcf","Type":"ContainerDied","Data":"8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c"}
Sep 05 03:00:06.349820 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:06.349784    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" event={"ID":"81680a36-5835-4bb1-beb0-1cef04afefcf","Type":"ContainerStarted","Data":"7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed"}
Sep 05 03:00:06.350310 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:06.349855    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" event={"ID":"81680a36-5835-4bb1-beb0-1cef04afefcf","Type":"ContainerStarted","Data":"e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c"}
Sep 05 03:00:06.350310 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:06.350054    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:06.376761 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:06.376700    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" podStartSLOduration=3.482949631 podStartE2EDuration="7.376678226s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.893728595s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:00:06.374291668 +0000 UTC m=+2527.382787830" watchObservedRunningTime="2026-09-05 03:00:06.376678226 +0000 UTC m=+2527.385174365"
Sep 05 03:00:07.356827 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:07.356789    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:07.358597 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:07.358556    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 05 03:00:08.360030 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:08.359992    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 05 03:00:13.365089 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:13.365055    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:13.366396 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:13.366373    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:19.833766 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:19.833734    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"]
Sep 05 03:00:19.834181 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:19.833981    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="kserve-container" containerID="cri-o://e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c" gracePeriod=30
Sep 05 03:00:19.834181 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:19.834010    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="kube-rbac-proxy" containerID="cri-o://7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed" gracePeriod=30
Sep 05 03:00:20.416549 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.416489    2555 generic.go:356] "Generic (PLEG): container finished" podID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerID="7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed" exitCode=2
Sep 05 03:00:20.416719 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.416588    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" event={"ID":"81680a36-5835-4bb1-beb0-1cef04afefcf","Type":"ContainerDied","Data":"7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed"}
Sep 05 03:00:20.572042 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.572015    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:20.692229 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.692192    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-rbac-proxy-sar-config\") pod \"81680a36-5835-4bb1-beb0-1cef04afefcf\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") "
Sep 05 03:00:20.692412 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.692249    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81680a36-5835-4bb1-beb0-1cef04afefcf-kserve-provision-location\") pod \"81680a36-5835-4bb1-beb0-1cef04afefcf\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") "
Sep 05 03:00:20.692412 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.692269    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81680a36-5835-4bb1-beb0-1cef04afefcf-proxy-tls\") pod \"81680a36-5835-4bb1-beb0-1cef04afefcf\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") "
Sep 05 03:00:20.692412 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.692297    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-878rx\" (UniqueName: \"kubernetes.io/projected/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-api-access-878rx\") pod \"81680a36-5835-4bb1-beb0-1cef04afefcf\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") "
Sep 05 03:00:20.692412 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.692315    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-cabundle-cert\") pod \"81680a36-5835-4bb1-beb0-1cef04afefcf\" (UID: \"81680a36-5835-4bb1-beb0-1cef04afefcf\") "
Sep 05 03:00:20.692721 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.692626    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81680a36-5835-4bb1-beb0-1cef04afefcf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81680a36-5835-4bb1-beb0-1cef04afefcf" (UID: "81680a36-5835-4bb1-beb0-1cef04afefcf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:20.692815 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.692720    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81680a36-5835-4bb1-beb0-1cef04afefcf" (UID: "81680a36-5835-4bb1-beb0-1cef04afefcf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:00:20.692861 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.692818    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81680a36-5835-4bb1-beb0-1cef04afefcf" (UID: "81680a36-5835-4bb1-beb0-1cef04afefcf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:00:20.694455 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.694431    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81680a36-5835-4bb1-beb0-1cef04afefcf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81680a36-5835-4bb1-beb0-1cef04afefcf" (UID: "81680a36-5835-4bb1-beb0-1cef04afefcf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:00:20.694582 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.694460    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-api-access-878rx" (OuterVolumeSpecName: "kube-api-access-878rx") pod "81680a36-5835-4bb1-beb0-1cef04afefcf" (UID: "81680a36-5835-4bb1-beb0-1cef04afefcf"). InnerVolumeSpecName "kube-api-access-878rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:00:20.793805 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.793770    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:00:20.793805 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.793802    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81680a36-5835-4bb1-beb0-1cef04afefcf-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:00:20.793805 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.793816    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81680a36-5835-4bb1-beb0-1cef04afefcf-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:00:20.794056 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.793831    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-878rx\" (UniqueName: \"kubernetes.io/projected/81680a36-5835-4bb1-beb0-1cef04afefcf-kube-api-access-878rx\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:00:20.794056 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:20.793846    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81680a36-5835-4bb1-beb0-1cef04afefcf-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:00:21.423470 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.423438    2555 generic.go:356] "Generic (PLEG): container finished" podID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerID="e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c" exitCode=0
Sep 05 03:00:21.423887 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.423496    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"
Sep 05 03:00:21.423887 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.423520    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" event={"ID":"81680a36-5835-4bb1-beb0-1cef04afefcf","Type":"ContainerDied","Data":"e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c"}
Sep 05 03:00:21.423887 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.423600    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj" event={"ID":"81680a36-5835-4bb1-beb0-1cef04afefcf","Type":"ContainerDied","Data":"8db93b7cd148eddb6a29a0e893553c98de25a740cf400d69fb4224e58af256f5"}
Sep 05 03:00:21.423887 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.423633    2555 scope.go:117] "RemoveContainer" containerID="7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed"
Sep 05 03:00:21.432877 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.432860    2555 scope.go:117] "RemoveContainer" containerID="e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c"
Sep 05 03:00:21.440107 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.440090    2555 scope.go:117] "RemoveContainer" containerID="8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c"
Sep 05 03:00:21.445755 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.445732    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"]
Sep 05 03:00:21.448817 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.448797    2555 scope.go:117] "RemoveContainer" containerID="7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed"
Sep 05 03:00:21.449118 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:00:21.449101    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed\": container with ID starting with 7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed not found: ID does not exist" containerID="7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed"
Sep 05 03:00:21.449189 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.449126    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed"} err="failed to get container status \"7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed\": rpc error: code = NotFound desc = could not find container \"7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed\": container with ID starting with 7a4ba929bb88c546438b5ad3086462e1a8540b83c3c44fe36139e0b0999d62ed not found: ID does not exist"
Sep 05 03:00:21.449189 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.449144    2555 scope.go:117] "RemoveContainer" containerID="e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c"
Sep 05 03:00:21.449412 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:00:21.449387    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c\": container with ID starting with e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c not found: ID does not exist" containerID="e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c"
Sep 05 03:00:21.449452 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.449417    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c"} err="failed to get container status \"e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c\": rpc error: code = NotFound desc = could not find container \"e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c\": container with ID starting with e51e8837d891770617c240308f4a1916dd1d680a13a73cc9b6fea48bb582ae6c not found: ID does not exist"
Sep 05 03:00:21.449452 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.449431    2555 scope.go:117] "RemoveContainer" containerID="8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c"
Sep 05 03:00:21.449566 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.449522    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tdklj"]
Sep 05 03:00:21.449712 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:00:21.449694    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c\": container with ID starting with 8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c not found: ID does not exist" containerID="8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c"
Sep 05 03:00:21.449758 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.449722    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c"} err="failed to get container status \"8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c\": rpc error: code = NotFound desc = could not find container \"8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c\": container with ID starting with 8f6f49384ecd6abb1ccc27d803d54937d1ef8c47a72de8d816b9a188f94aef2c not found: ID does not exist"
Sep 05 03:00:21.543268 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:21.543235    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" path="/var/lib/kubelet/pods/81680a36-5835-4bb1-beb0-1cef04afefcf/volumes"
Sep 05 03:00:25.562191 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.562156    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"]
Sep 05 03:00:25.562623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.562479    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="storage-initializer"
Sep 05 03:00:25.562623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.562490    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="storage-initializer"
Sep 05 03:00:25.562623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.562508    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="kserve-container"
Sep 05 03:00:25.562623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.562513    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="kserve-container"
Sep 05 03:00:25.562623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.562528    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="kube-rbac-proxy"
Sep 05 03:00:25.562623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.562546    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="kube-rbac-proxy"
Sep 05 03:00:25.562623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.562613    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="kserve-container"
Sep 05 03:00:25.562623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.562623    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="81680a36-5835-4bb1-beb0-1cef04afefcf" containerName="kube-rbac-proxy"
Sep 05 03:00:25.566956 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.566934    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.569272 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.569241    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 05 03:00:25.569272 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.569255    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 03:00:25.569471 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.569304    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:00:25.569471 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.569356    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 03:00:25.569606 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.569527    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 05 03:00:25.570171 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.570156    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 03:00:25.570266 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.570216    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 03:00:25.575886 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.575861    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"]
Sep 05 03:00:25.737917 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.737879    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce8ee1e8-f313-418e-a9ee-b30eb0385704-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.737917 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.737917    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm4p7\" (UniqueName: \"kubernetes.io/projected/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-api-access-fm4p7\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.738196 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.737942    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.738196 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.738038    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.738196 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.738105    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.839178 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.839081    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce8ee1e8-f313-418e-a9ee-b30eb0385704-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.839178 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.839121    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm4p7\" (UniqueName: \"kubernetes.io/projected/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-api-access-fm4p7\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.839178 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.839143    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.839178 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.839175    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.839477 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.839232    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.839726 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.839700    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.840047 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.840023    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.840096 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.840023    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.841582 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.841563    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce8ee1e8-f313-418e-a9ee-b30eb0385704-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.847080 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.847058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm4p7\" (UniqueName: \"kubernetes.io/projected/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-api-access-fm4p7\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ql72m\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:25.878137 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:25.878098    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:26.151446 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:26.151420    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"]
Sep 05 03:00:26.153598 ip-10-0-135-239 kubenswrapper[2555]: W0905 03:00:26.153570    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8ee1e8_f313_418e_a9ee_b30eb0385704.slice/crio-2c985dcac9028a83cbd38f9b4bfd0bb01bfbdc829642225f89fbd792273edc1a WatchSource:0}: Error finding container 2c985dcac9028a83cbd38f9b4bfd0bb01bfbdc829642225f89fbd792273edc1a: Status 404 returned error can't find the container with id 2c985dcac9028a83cbd38f9b4bfd0bb01bfbdc829642225f89fbd792273edc1a
Sep 05 03:00:26.447524 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:26.447435    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" event={"ID":"ce8ee1e8-f313-418e-a9ee-b30eb0385704","Type":"ContainerStarted","Data":"73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3"}
Sep 05 03:00:26.447685 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:26.447554    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" event={"ID":"ce8ee1e8-f313-418e-a9ee-b30eb0385704","Type":"ContainerStarted","Data":"2c985dcac9028a83cbd38f9b4bfd0bb01bfbdc829642225f89fbd792273edc1a"}
Sep 05 03:00:30.466851 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:30.466819    2555 generic.go:356] "Generic (PLEG): container finished" podID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerID="73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3" exitCode=0
Sep 05 03:00:30.467208 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:30.466888    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" event={"ID":"ce8ee1e8-f313-418e-a9ee-b30eb0385704","Type":"ContainerDied","Data":"73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3"}
Sep 05 03:00:31.473952 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:31.473915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" event={"ID":"ce8ee1e8-f313-418e-a9ee-b30eb0385704","Type":"ContainerStarted","Data":"37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3"}
Sep 05 03:00:31.474403 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:31.474004    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" event={"ID":"ce8ee1e8-f313-418e-a9ee-b30eb0385704","Type":"ContainerStarted","Data":"bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040"}
Sep 05 03:00:31.474403 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:31.474186    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:31.492871 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:31.492822    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podStartSLOduration=2.429759519 podStartE2EDuration="6.492806134s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.063046615s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:00:31.490662143 +0000 UTC m=+2552.499158301" watchObservedRunningTime="2026-09-05 03:00:31.492806134 +0000 UTC m=+2552.501302272"
Sep 05 03:00:32.478674 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:32.478641    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:32.479975 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:32.479948    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 05 03:00:33.483092 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:33.483047    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 05 03:00:38.487848 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:38.487818    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:00:38.488328 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:38.488302    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 05 03:00:48.488310 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:48.488274    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 05 03:00:58.489263 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:00:58.489218    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 05 03:01:08.488663 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:08.488621    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 05 03:01:18.488503 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:18.488460    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 05 03:01:20.269181 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.269144    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"]
Sep 05 03:01:20.272825 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.272805    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.275077 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.275057    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 05 03:01:20.275162 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.275098    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 05 03:01:20.283938 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.283913    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"]
Sep 05 03:01:20.315431 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.315395    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fae2d81-cc91-4962-baf4-f670c0788b1d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.315608 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.315470    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fae2d81-cc91-4962-baf4-f670c0788b1d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.315608 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.315499    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkb54\" (UniqueName: \"kubernetes.io/projected/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-api-access-gkb54\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.315608 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.315521    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.315608 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.315571    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.416562 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.416496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fae2d81-cc91-4962-baf4-f670c0788b1d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.416763 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.416580    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkb54\" (UniqueName: \"kubernetes.io/projected/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-api-access-gkb54\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.416763 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.416614    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.416763 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.416643    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.416763 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.416696    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fae2d81-cc91-4962-baf4-f670c0788b1d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.416986 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:01:20.416846    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 05 03:01:20.416986 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.416883    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fae2d81-cc91-4962-baf4-f670c0788b1d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.416986 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:01:20.416931    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fae2d81-cc91-4962-baf4-f670c0788b1d-proxy-tls podName:4fae2d81-cc91-4962-baf4-f670c0788b1d nodeName:}" failed. No retries permitted until 2026-09-05 03:01:20.916908166 +0000 UTC m=+2601.925404286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4fae2d81-cc91-4962-baf4-f670c0788b1d-proxy-tls") pod "isvc-sklearn-mcp-predictor-8649d89897-545k6" (UID: "4fae2d81-cc91-4962-baf4-f670c0788b1d") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 05 03:01:20.417229 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.417211    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.417332 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.417314    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.425006 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.424987    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkb54\" (UniqueName: \"kubernetes.io/projected/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-api-access-gkb54\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.920974 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.920938    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fae2d81-cc91-4962-baf4-f670c0788b1d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:20.923275 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:20.923245    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fae2d81-cc91-4962-baf4-f670c0788b1d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-8649d89897-545k6\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:21.183021 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:21.182924    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:21.456884 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:21.456808    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"]
Sep 05 03:01:21.460256 ip-10-0-135-239 kubenswrapper[2555]: W0905 03:01:21.460223    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fae2d81_cc91_4962_baf4_f670c0788b1d.slice/crio-821317953cdecc485aad4ad220fb38b02f81f484b9918162448a202108a829c9 WatchSource:0}: Error finding container 821317953cdecc485aad4ad220fb38b02f81f484b9918162448a202108a829c9: Status 404 returned error can't find the container with id 821317953cdecc485aad4ad220fb38b02f81f484b9918162448a202108a829c9
Sep 05 03:01:21.713672 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:21.713581    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" event={"ID":"4fae2d81-cc91-4962-baf4-f670c0788b1d","Type":"ContainerStarted","Data":"429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e"}
Sep 05 03:01:21.713672 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:21.713656    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" event={"ID":"4fae2d81-cc91-4962-baf4-f670c0788b1d","Type":"ContainerStarted","Data":"821317953cdecc485aad4ad220fb38b02f81f484b9918162448a202108a829c9"}
Sep 05 03:01:25.734703 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:25.734664    2555 generic.go:356] "Generic (PLEG): container finished" podID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerID="429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e" exitCode=0
Sep 05 03:01:25.735122 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:25.734742    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" event={"ID":"4fae2d81-cc91-4962-baf4-f670c0788b1d","Type":"ContainerDied","Data":"429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e"}
Sep 05 03:01:25.825495 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:25.825471    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 03:01:26.742438 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:26.742405    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" event={"ID":"4fae2d81-cc91-4962-baf4-f670c0788b1d","Type":"ContainerStarted","Data":"bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9"}
Sep 05 03:01:28.488379 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:28.488341    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 05 03:01:29.759612 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:29.759573    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" event={"ID":"4fae2d81-cc91-4962-baf4-f670c0788b1d","Type":"ContainerStarted","Data":"37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8"}
Sep 05 03:01:29.760039 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:29.759640    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" event={"ID":"4fae2d81-cc91-4962-baf4-f670c0788b1d","Type":"ContainerStarted","Data":"0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca"}
Sep 05 03:01:29.760039 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:29.759787    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:29.760039 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:29.759829    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:29.760039 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:29.759852    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:29.782317 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:29.782273    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podStartSLOduration=2.5430587190000002 podStartE2EDuration="9.782258133s" totalImagesPullingTime="3.236906773s" totalInitContainerRuntime="4.002292641s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:01:29.78123257 +0000 UTC m=+2610.789728728" watchObservedRunningTime="2026-09-05 03:01:29.782258133 +0000 UTC m=+2610.790754272"
Sep 05 03:01:35.769655 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:35.769626    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:01:38.489711 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:38.489675    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:01:45.664414 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:45.664380    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"]
Sep 05 03:01:45.664846 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:45.664641    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container" containerID="cri-o://bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040" gracePeriod=30
Sep 05 03:01:45.664846 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:45.664724    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kube-rbac-proxy" containerID="cri-o://37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3" gracePeriod=30
Sep 05 03:01:45.832604 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:45.832563    2555 generic.go:356] "Generic (PLEG): container finished" podID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerID="37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3" exitCode=2
Sep 05 03:01:45.832604 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:45.832609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" event={"ID":"ce8ee1e8-f313-418e-a9ee-b30eb0385704","Type":"ContainerDied","Data":"37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3"}
Sep 05 03:01:48.483898 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:48.483854    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Sep 05 03:01:48.488873 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:48.488840    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 05 03:01:49.410450 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.410426    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:01:49.577669 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.577587    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-rbac-proxy-sar-config\") pod \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") "
Sep 05 03:01:49.577669 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.577626    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kserve-provision-location\") pod \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") "
Sep 05 03:01:49.578139 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.577675    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce8ee1e8-f313-418e-a9ee-b30eb0385704-proxy-tls\") pod \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") "
Sep 05 03:01:49.578139 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.577720    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-cabundle-cert\") pod \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") "
Sep 05 03:01:49.578139 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.577766    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm4p7\" (UniqueName: \"kubernetes.io/projected/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-api-access-fm4p7\") pod \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\" (UID: \"ce8ee1e8-f313-418e-a9ee-b30eb0385704\") "
Sep 05 03:01:49.578139 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.577971    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce8ee1e8-f313-418e-a9ee-b30eb0385704" (UID: "ce8ee1e8-f313-418e-a9ee-b30eb0385704"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:01:49.578139 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.578033    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce8ee1e8-f313-418e-a9ee-b30eb0385704" (UID: "ce8ee1e8-f313-418e-a9ee-b30eb0385704"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:01:49.578139 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.578083    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:01:49.578139 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.578128    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce8ee1e8-f313-418e-a9ee-b30eb0385704" (UID: "ce8ee1e8-f313-418e-a9ee-b30eb0385704"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:01:49.579845 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.579819    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce8ee1e8-f313-418e-a9ee-b30eb0385704-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce8ee1e8-f313-418e-a9ee-b30eb0385704" (UID: "ce8ee1e8-f313-418e-a9ee-b30eb0385704"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:01:49.579944 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.579861    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-api-access-fm4p7" (OuterVolumeSpecName: "kube-api-access-fm4p7") pod "ce8ee1e8-f313-418e-a9ee-b30eb0385704" (UID: "ce8ee1e8-f313-418e-a9ee-b30eb0385704"). InnerVolumeSpecName "kube-api-access-fm4p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:01:49.679102 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.679062    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:01:49.679102 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.679094    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm4p7\" (UniqueName: \"kubernetes.io/projected/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-api-access-fm4p7\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:01:49.679102 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.679107    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce8ee1e8-f313-418e-a9ee-b30eb0385704-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:01:49.679329 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.679121    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce8ee1e8-f313-418e-a9ee-b30eb0385704-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:01:49.855724 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.855640    2555 generic.go:356] "Generic (PLEG): container finished" podID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerID="bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040" exitCode=0
Sep 05 03:01:49.855724 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.855678    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" event={"ID":"ce8ee1e8-f313-418e-a9ee-b30eb0385704","Type":"ContainerDied","Data":"bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040"}
Sep 05 03:01:49.855724 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.855700    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"
Sep 05 03:01:49.855724 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.855724    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m" event={"ID":"ce8ee1e8-f313-418e-a9ee-b30eb0385704","Type":"ContainerDied","Data":"2c985dcac9028a83cbd38f9b4bfd0bb01bfbdc829642225f89fbd792273edc1a"}
Sep 05 03:01:49.856059 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.855750    2555 scope.go:117] "RemoveContainer" containerID="37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3"
Sep 05 03:01:49.864409 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.864390    2555 scope.go:117] "RemoveContainer" containerID="bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040"
Sep 05 03:01:49.872284 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.872264    2555 scope.go:117] "RemoveContainer" containerID="73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3"
Sep 05 03:01:49.878437 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.878415    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"]
Sep 05 03:01:49.879680 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.879659    2555 scope.go:117] "RemoveContainer" containerID="37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3"
Sep 05 03:01:49.879940 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:01:49.879925    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3\": container with ID starting with 37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3 not found: ID does not exist" containerID="37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3"
Sep 05 03:01:49.879993 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.879948    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3"} err="failed to get container status \"37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3\": rpc error: code = NotFound desc = could not find container \"37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3\": container with ID starting with 37836e8634d9fe2ecae8b564c058e6141973d7da2cfaa4b5209684bdd4372fc3 not found: ID does not exist"
Sep 05 03:01:49.879993 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.879965    2555 scope.go:117] "RemoveContainer" containerID="bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040"
Sep 05 03:01:49.880303 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:01:49.880278    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040\": container with ID starting with bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040 not found: ID does not exist" containerID="bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040"
Sep 05 03:01:49.880403 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.880309    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040"} err="failed to get container status \"bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040\": rpc error: code = NotFound desc = could not find container \"bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040\": container with ID starting with bb21ee215b2166534ce1890f11fe071ca3cfb74ab06a1cdf41131cd31d3f1040 not found: ID does not exist"
Sep 05 03:01:49.880403 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.880330    2555 scope.go:117] "RemoveContainer" containerID="73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3"
Sep 05 03:01:49.880821 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:01:49.880791    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3\": container with ID starting with 73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3 not found: ID does not exist" containerID="73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3"
Sep 05 03:01:49.880934 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.880829    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3"} err="failed to get container status \"73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3\": rpc error: code = NotFound desc = could not find container \"73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3\": container with ID starting with 73a705ec9747962efa9769b515f366bdbfa65dbeb6337e830df4c5faf03e10a3 not found: ID does not exist"
Sep 05 03:01:49.882631 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:49.882610    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ql72m"]
Sep 05 03:01:50.971628 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.971591    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"]
Sep 05 03:01:50.972031 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.971926    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kube-rbac-proxy"
Sep 05 03:01:50.972031 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.971937    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kube-rbac-proxy"
Sep 05 03:01:50.972031 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.971963    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="storage-initializer"
Sep 05 03:01:50.972031 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.971969    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="storage-initializer"
Sep 05 03:01:50.972031 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.971975    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container"
Sep 05 03:01:50.972031 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.971980    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container"
Sep 05 03:01:50.972031 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.972029    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kube-rbac-proxy"
Sep 05 03:01:50.972288 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.972038    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" containerName="kserve-container"
Sep 05 03:01:50.976685 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.976667    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:50.979740 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.979717    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 05 03:01:50.979889 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.979720    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 05 03:01:50.979889 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.979724    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 05 03:01:50.990030 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:50.990004    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"]
Sep 05 03:01:51.089552 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.089496    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37f82e54-600c-4823-8b77-9d2385aa814d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.089738 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.089572    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f82e54-600c-4823-8b77-9d2385aa814d-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.089738 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.089645    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brrz4\" (UniqueName: \"kubernetes.io/projected/37f82e54-600c-4823-8b77-9d2385aa814d-kube-api-access-brrz4\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.089738 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.089691    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f82e54-600c-4823-8b77-9d2385aa814d-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.190701 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.190668    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f82e54-600c-4823-8b77-9d2385aa814d-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.190863 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.190735    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brrz4\" (UniqueName: \"kubernetes.io/projected/37f82e54-600c-4823-8b77-9d2385aa814d-kube-api-access-brrz4\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.190863 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.190783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f82e54-600c-4823-8b77-9d2385aa814d-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.190863 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.190823    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37f82e54-600c-4823-8b77-9d2385aa814d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.190863 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:01:51.190827    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 05 03:01:51.191079 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:01:51.190896    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37f82e54-600c-4823-8b77-9d2385aa814d-proxy-tls podName:37f82e54-600c-4823-8b77-9d2385aa814d nodeName:}" failed. No retries permitted until 2026-09-05 03:01:51.690872538 +0000 UTC m=+2632.699368677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/37f82e54-600c-4823-8b77-9d2385aa814d-proxy-tls") pod "isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" (UID: "37f82e54-600c-4823-8b77-9d2385aa814d") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 05 03:01:51.191243 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.191219    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f82e54-600c-4823-8b77-9d2385aa814d-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.191474 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.191455    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37f82e54-600c-4823-8b77-9d2385aa814d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.199208 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.199187    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brrz4\" (UniqueName: \"kubernetes.io/projected/37f82e54-600c-4823-8b77-9d2385aa814d-kube-api-access-brrz4\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.550432 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.550387    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce8ee1e8-f313-418e-a9ee-b30eb0385704" path="/var/lib/kubelet/pods/ce8ee1e8-f313-418e-a9ee-b30eb0385704/volumes"
Sep 05 03:01:51.696665 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.696626    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f82e54-600c-4823-8b77-9d2385aa814d-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.699071 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.699049    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f82e54-600c-4823-8b77-9d2385aa814d-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7dbcbf9746-qnc59\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:51.886884 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:51.886844    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:52.161647 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:52.161623    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"]
Sep 05 03:01:52.163618 ip-10-0-135-239 kubenswrapper[2555]: W0905 03:01:52.163586    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f82e54_600c_4823_8b77_9d2385aa814d.slice/crio-87747a4a35f25f55df4cca853b1fb1582136eda98e6d601d81965c35b71e78e4 WatchSource:0}: Error finding container 87747a4a35f25f55df4cca853b1fb1582136eda98e6d601d81965c35b71e78e4: Status 404 returned error can't find the container with id 87747a4a35f25f55df4cca853b1fb1582136eda98e6d601d81965c35b71e78e4
Sep 05 03:01:52.872042 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:52.871998    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" event={"ID":"37f82e54-600c-4823-8b77-9d2385aa814d","Type":"ContainerStarted","Data":"dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9"}
Sep 05 03:01:52.872236 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:52.872082    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" event={"ID":"37f82e54-600c-4823-8b77-9d2385aa814d","Type":"ContainerStarted","Data":"87747a4a35f25f55df4cca853b1fb1582136eda98e6d601d81965c35b71e78e4"}
Sep 05 03:01:53.877892 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:53.877856    2555 generic.go:356] "Generic (PLEG): container finished" podID="37f82e54-600c-4823-8b77-9d2385aa814d" containerID="dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9" exitCode=0
Sep 05 03:01:53.878263 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:53.877907    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" event={"ID":"37f82e54-600c-4823-8b77-9d2385aa814d","Type":"ContainerDied","Data":"dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9"}
Sep 05 03:01:54.884792 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:54.884756    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" event={"ID":"37f82e54-600c-4823-8b77-9d2385aa814d","Type":"ContainerStarted","Data":"c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74"}
Sep 05 03:01:54.885283 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:54.884824    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" event={"ID":"37f82e54-600c-4823-8b77-9d2385aa814d","Type":"ContainerStarted","Data":"53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47"}
Sep 05 03:01:54.885283 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:54.884863    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:54.904432 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:54.904377    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podStartSLOduration=3.86429795 podStartE2EDuration="4.904359377s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.040061427s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:01:54.903476508 +0000 UTC m=+2635.911972678" watchObservedRunningTime="2026-09-05 03:01:54.904359377 +0000 UTC m=+2635.912855581"
Sep 05 03:01:55.770930 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:55.770888    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 05 03:01:55.889598 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:55.889566    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:01:55.891141 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:55.891103    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 05 03:01:56.893818 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:01:56.893775    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 05 03:02:01.183382 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:01.183326    2555 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 05 03:02:01.898862 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:01.898833    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:02:01.899336 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:01.899308    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 05 03:02:05.774616 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:05.774582    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:02:11.900028 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:11.899987    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 05 03:02:21.899996 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:21.899949    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 05 03:02:31.899668 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:31.899625    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 05 03:02:35.776263 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:35.776233    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:02:40.353669 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:40.353633    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"]
Sep 05 03:02:40.354214 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:40.353959    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-container" containerID="cri-o://bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9" gracePeriod=30
Sep 05 03:02:40.354214 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:40.354029    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-agent" containerID="cri-o://0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca" gracePeriod=30
Sep 05 03:02:40.354214 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:40.354035    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kube-rbac-proxy" containerID="cri-o://37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8" gracePeriod=30
Sep 05 03:02:40.764484 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:40.764434    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 05 03:02:41.095371 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:41.095284    2555 generic.go:356] "Generic (PLEG): container finished" podID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerID="37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8" exitCode=2
Sep 05 03:02:41.095371 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:41.095354    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" event={"ID":"4fae2d81-cc91-4962-baf4-f670c0788b1d","Type":"ContainerDied","Data":"37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8"}
Sep 05 03:02:41.899957 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:41.899918    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 05 03:02:43.107075 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:43.107040    2555 generic.go:356] "Generic (PLEG): container finished" podID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerID="bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9" exitCode=0
Sep 05 03:02:43.107468 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:43.107117    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" event={"ID":"4fae2d81-cc91-4962-baf4-f670c0788b1d","Type":"ContainerDied","Data":"bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9"}
Sep 05 03:02:45.764923 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:45.764879    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 05 03:02:45.770320 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:45.770294    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 05 03:02:50.764723 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:50.764677    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 05 03:02:50.765160 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:50.764818    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:02:51.900281 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:51.900199    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 05 03:02:55.764662 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:55.764614    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 05 03:02:55.770856 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:55.770821    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 05 03:02:59.678912 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:59.678886    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 03:02:59.680870 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:02:59.680846    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 03:03:00.764831 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:00.764787    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 05 03:03:01.899763 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:01.899723    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 05 03:03:05.764992 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:05.764950    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 05 03:03:05.770377 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:05.770342    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 05 03:03:05.770517 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:05.770498    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:03:10.533210 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.533184    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:03:10.577303 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.577273    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fae2d81-cc91-4962-baf4-f670c0788b1d-proxy-tls\") pod \"4fae2d81-cc91-4962-baf4-f670c0788b1d\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") "
Sep 05 03:03:10.577487 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.577362    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fae2d81-cc91-4962-baf4-f670c0788b1d-kserve-provision-location\") pod \"4fae2d81-cc91-4962-baf4-f670c0788b1d\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") "
Sep 05 03:03:10.577487 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.577428    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkb54\" (UniqueName: \"kubernetes.io/projected/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-api-access-gkb54\") pod \"4fae2d81-cc91-4962-baf4-f670c0788b1d\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") "
Sep 05 03:03:10.577487 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.577467    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-rbac-proxy-sar-config\") pod \"4fae2d81-cc91-4962-baf4-f670c0788b1d\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") "
Sep 05 03:03:10.577678 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.577501    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-cabundle-cert\") pod \"4fae2d81-cc91-4962-baf4-f670c0788b1d\" (UID: \"4fae2d81-cc91-4962-baf4-f670c0788b1d\") "
Sep 05 03:03:10.577730 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.577695    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fae2d81-cc91-4962-baf4-f670c0788b1d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4fae2d81-cc91-4962-baf4-f670c0788b1d" (UID: "4fae2d81-cc91-4962-baf4-f670c0788b1d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:03:10.577872 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.577844    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4fae2d81-cc91-4962-baf4-f670c0788b1d" (UID: "4fae2d81-cc91-4962-baf4-f670c0788b1d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:03:10.577872 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.577863    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fae2d81-cc91-4962-baf4-f670c0788b1d-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:03:10.577976 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.577904    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4fae2d81-cc91-4962-baf4-f670c0788b1d" (UID: "4fae2d81-cc91-4962-baf4-f670c0788b1d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:03:10.579502 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.579477    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-api-access-gkb54" (OuterVolumeSpecName: "kube-api-access-gkb54") pod "4fae2d81-cc91-4962-baf4-f670c0788b1d" (UID: "4fae2d81-cc91-4962-baf4-f670c0788b1d"). InnerVolumeSpecName "kube-api-access-gkb54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:03:10.579502 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.579501    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fae2d81-cc91-4962-baf4-f670c0788b1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4fae2d81-cc91-4962-baf4-f670c0788b1d" (UID: "4fae2d81-cc91-4962-baf4-f670c0788b1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:03:10.679231 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.679192    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkb54\" (UniqueName: \"kubernetes.io/projected/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-api-access-gkb54\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:03:10.679231 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.679223    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:03:10.679231 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.679232    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fae2d81-cc91-4962-baf4-f670c0788b1d-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:03:10.679493 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:10.679245    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fae2d81-cc91-4962-baf4-f670c0788b1d-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:03:11.238364 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.238325    2555 generic.go:356] "Generic (PLEG): container finished" podID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerID="0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca" exitCode=137
Sep 05 03:03:11.238552 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.238393    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"
Sep 05 03:03:11.238552 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.238406    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" event={"ID":"4fae2d81-cc91-4962-baf4-f670c0788b1d","Type":"ContainerDied","Data":"0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca"}
Sep 05 03:03:11.238552 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.238481    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6" event={"ID":"4fae2d81-cc91-4962-baf4-f670c0788b1d","Type":"ContainerDied","Data":"821317953cdecc485aad4ad220fb38b02f81f484b9918162448a202108a829c9"}
Sep 05 03:03:11.238552 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.238516    2555 scope.go:117] "RemoveContainer" containerID="37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8"
Sep 05 03:03:11.249464 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.249441    2555 scope.go:117] "RemoveContainer" containerID="0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca"
Sep 05 03:03:11.257509 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.257486    2555 scope.go:117] "RemoveContainer" containerID="bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9"
Sep 05 03:03:11.262917 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.262892    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"]
Sep 05 03:03:11.266049 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.266001    2555 scope.go:117] "RemoveContainer" containerID="429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e"
Sep 05 03:03:11.268250 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.268230    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8649d89897-545k6"]
Sep 05 03:03:11.274125 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.274108    2555 scope.go:117] "RemoveContainer" containerID="37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8"
Sep 05 03:03:11.274368 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:03:11.274351    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8\": container with ID starting with 37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8 not found: ID does not exist" containerID="37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8"
Sep 05 03:03:11.274426 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.274376    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8"} err="failed to get container status \"37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8\": rpc error: code = NotFound desc = could not find container \"37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8\": container with ID starting with 37056766aa36b707ba217f259783ec5f96cf32426530539a9b254998a1cfeef8 not found: ID does not exist"
Sep 05 03:03:11.274426 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.274404    2555 scope.go:117] "RemoveContainer" containerID="0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca"
Sep 05 03:03:11.274673 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:03:11.274656    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca\": container with ID starting with 0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca not found: ID does not exist" containerID="0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca"
Sep 05 03:03:11.274723 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.274677    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca"} err="failed to get container status \"0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca\": rpc error: code = NotFound desc = could not find container \"0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca\": container with ID starting with 0968336da4fd7d36539c4bb806648fb17cf1c0b3a8c50b86faa6f2d0015baaca not found: ID does not exist"
Sep 05 03:03:11.274723 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.274690    2555 scope.go:117] "RemoveContainer" containerID="bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9"
Sep 05 03:03:11.274922 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:03:11.274907    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9\": container with ID starting with bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9 not found: ID does not exist" containerID="bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9"
Sep 05 03:03:11.274963 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.274926    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9"} err="failed to get container status \"bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9\": rpc error: code = NotFound desc = could not find container \"bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9\": container with ID starting with bb54253e08fdc987f126dfb2ce2dd1b2bf1ba510507ff4824e72593fcb84c2d9 not found: ID does not exist"
Sep 05 03:03:11.274963 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.274940    2555 scope.go:117] "RemoveContainer" containerID="429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e"
Sep 05 03:03:11.275140 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:03:11.275126    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e\": container with ID starting with 429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e not found: ID does not exist" containerID="429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e"
Sep 05 03:03:11.275180 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.275141    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e"} err="failed to get container status \"429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e\": rpc error: code = NotFound desc = could not find container \"429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e\": container with ID starting with 429bcd7b9dfbf8828f71fe2fa1b2b8656480dff1ebe329e58c46fcc8f9e5d71e not found: ID does not exist"
Sep 05 03:03:11.544267 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.544187    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" path="/var/lib/kubelet/pods/4fae2d81-cc91-4962-baf4-f670c0788b1d/volumes"
Sep 05 03:03:11.900629 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:11.900601    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:03:21.059931 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:21.059896    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"]
Sep 05 03:03:21.060345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:21.060130    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container" containerID="cri-o://53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47" gracePeriod=30
Sep 05 03:03:21.060345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:21.060170    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kube-rbac-proxy" containerID="cri-o://c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74" gracePeriod=30
Sep 05 03:03:21.295092 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:21.295062    2555 generic.go:356] "Generic (PLEG): container finished" podID="37f82e54-600c-4823-8b77-9d2385aa814d" containerID="c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74" exitCode=2
Sep 05 03:03:21.295272 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:21.295110    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" event={"ID":"37f82e54-600c-4823-8b77-9d2385aa814d","Type":"ContainerDied","Data":"c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74"}
Sep 05 03:03:21.894491 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:21.894440    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 05 03:03:21.899677 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:21.899647    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 05 03:03:25.806987 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:25.806962    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:03:25.910961 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:25.910876    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brrz4\" (UniqueName: \"kubernetes.io/projected/37f82e54-600c-4823-8b77-9d2385aa814d-kube-api-access-brrz4\") pod \"37f82e54-600c-4823-8b77-9d2385aa814d\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") "
Sep 05 03:03:25.910961 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:25.910929    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f82e54-600c-4823-8b77-9d2385aa814d-kserve-provision-location\") pod \"37f82e54-600c-4823-8b77-9d2385aa814d\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") "
Sep 05 03:03:25.911191 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:25.910966    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f82e54-600c-4823-8b77-9d2385aa814d-proxy-tls\") pod \"37f82e54-600c-4823-8b77-9d2385aa814d\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") "
Sep 05 03:03:25.911191 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:25.911020    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37f82e54-600c-4823-8b77-9d2385aa814d-kube-rbac-proxy-sar-config\") pod \"37f82e54-600c-4823-8b77-9d2385aa814d\" (UID: \"37f82e54-600c-4823-8b77-9d2385aa814d\") "
Sep 05 03:03:25.911324 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:25.911301    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f82e54-600c-4823-8b77-9d2385aa814d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37f82e54-600c-4823-8b77-9d2385aa814d" (UID: "37f82e54-600c-4823-8b77-9d2385aa814d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:03:25.911428 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:25.911404    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f82e54-600c-4823-8b77-9d2385aa814d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37f82e54-600c-4823-8b77-9d2385aa814d" (UID: "37f82e54-600c-4823-8b77-9d2385aa814d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:03:25.912996 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:25.912971    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f82e54-600c-4823-8b77-9d2385aa814d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37f82e54-600c-4823-8b77-9d2385aa814d" (UID: "37f82e54-600c-4823-8b77-9d2385aa814d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:03:25.912996 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:25.912969    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f82e54-600c-4823-8b77-9d2385aa814d-kube-api-access-brrz4" (OuterVolumeSpecName: "kube-api-access-brrz4") pod "37f82e54-600c-4823-8b77-9d2385aa814d" (UID: "37f82e54-600c-4823-8b77-9d2385aa814d"). InnerVolumeSpecName "kube-api-access-brrz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:03:26.012456 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.012406    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brrz4\" (UniqueName: \"kubernetes.io/projected/37f82e54-600c-4823-8b77-9d2385aa814d-kube-api-access-brrz4\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:03:26.012456 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.012450    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f82e54-600c-4823-8b77-9d2385aa814d-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:03:26.012456 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.012462    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37f82e54-600c-4823-8b77-9d2385aa814d-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:03:26.012456 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.012471    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37f82e54-600c-4823-8b77-9d2385aa814d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:03:26.319459 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.319355    2555 generic.go:356] "Generic (PLEG): container finished" podID="37f82e54-600c-4823-8b77-9d2385aa814d" containerID="53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47" exitCode=0
Sep 05 03:03:26.319459 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.319414    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"
Sep 05 03:03:26.319739 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.319417    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" event={"ID":"37f82e54-600c-4823-8b77-9d2385aa814d","Type":"ContainerDied","Data":"53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47"}
Sep 05 03:03:26.319739 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.319581    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59" event={"ID":"37f82e54-600c-4823-8b77-9d2385aa814d","Type":"ContainerDied","Data":"87747a4a35f25f55df4cca853b1fb1582136eda98e6d601d81965c35b71e78e4"}
Sep 05 03:03:26.319739 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.319623    2555 scope.go:117] "RemoveContainer" containerID="c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74"
Sep 05 03:03:26.328934 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.328919    2555 scope.go:117] "RemoveContainer" containerID="53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47"
Sep 05 03:03:26.336685 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.336665    2555 scope.go:117] "RemoveContainer" containerID="dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9"
Sep 05 03:03:26.347351 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.347304    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"]
Sep 05 03:03:26.347647 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.347621    2555 scope.go:117] "RemoveContainer" containerID="c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74"
Sep 05 03:03:26.347747 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.347683    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7dbcbf9746-qnc59"]
Sep 05 03:03:26.348100 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:03:26.348076    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74\": container with ID starting with c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74 not found: ID does not exist" containerID="c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74"
Sep 05 03:03:26.348187 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.348107    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74"} err="failed to get container status \"c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74\": rpc error: code = NotFound desc = could not find container \"c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74\": container with ID starting with c833b4d718d897140bb5623c5f67935dbc5c9e23c42789174ca888b6ded8ab74 not found: ID does not exist"
Sep 05 03:03:26.348187 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.348126    2555 scope.go:117] "RemoveContainer" containerID="53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47"
Sep 05 03:03:26.348435 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:03:26.348411    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47\": container with ID starting with 53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47 not found: ID does not exist" containerID="53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47"
Sep 05 03:03:26.348555 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.348439    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47"} err="failed to get container status \"53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47\": rpc error: code = NotFound desc = could not find container \"53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47\": container with ID starting with 53946d19070e0c82b754eacd332ef32869c081790cbcbef8f0e1ddbb2aa98c47 not found: ID does not exist"
Sep 05 03:03:26.348555 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.348455    2555 scope.go:117] "RemoveContainer" containerID="dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9"
Sep 05 03:03:26.348752 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:03:26.348732    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9\": container with ID starting with dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9 not found: ID does not exist" containerID="dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9"
Sep 05 03:03:26.348811 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:26.348761    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9"} err="failed to get container status \"dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9\": rpc error: code = NotFound desc = could not find container \"dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9\": container with ID starting with dd6ae0a0518ebc5385752b36e528512967c0c1b8bb701324192730179b0462d9 not found: ID does not exist"
Sep 05 03:03:27.544130 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:03:27.544091    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" path="/var/lib/kubelet/pods/37f82e54-600c-4823-8b77-9d2385aa814d/volumes"
Sep 05 03:04:01.762858 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.762820    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"]
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763150    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-container"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763162    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-container"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763169    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kube-rbac-proxy"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763174    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kube-rbac-proxy"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763180    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kube-rbac-proxy"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763186    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kube-rbac-proxy"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763197    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-agent"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763202    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-agent"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763209    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="storage-initializer"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763214    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="storage-initializer"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763221    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="storage-initializer"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763226    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="storage-initializer"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763235    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763240    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763287    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kube-rbac-proxy"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763294    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kube-rbac-proxy"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763300    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-agent"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763306    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f82e54-600c-4823-8b77-9d2385aa814d" containerName="kserve-container"
Sep 05 03:04:01.763345 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.763313    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fae2d81-cc91-4962-baf4-f670c0788b1d" containerName="kserve-container"
Sep 05 03:04:01.767853 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.767835    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.770982 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.770954    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 03:04:01.770982 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.770965    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 03:04:01.771165 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.770982    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 05 03:04:01.771165 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.770968    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 05 03:04:01.771165 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.771024    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 03:04:01.771826 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.771808    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 03:04:01.771947 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.771843    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:04:01.776597 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.776575    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"]
Sep 05 03:04:01.797175 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.797144    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.797317 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.797180    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6515eee-e003-4a20-a44e-1d514def0ddb-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.797317 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.797199    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6515eee-e003-4a20-a44e-1d514def0ddb-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.797317 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.797284    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.797317 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.797302    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxpk2\" (UniqueName: \"kubernetes.io/projected/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-api-access-zxpk2\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.898405 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.898371    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.898589 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.898413    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6515eee-e003-4a20-a44e-1d514def0ddb-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.898589 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.898431    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6515eee-e003-4a20-a44e-1d514def0ddb-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.898589 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.898577    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.898776 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.898608    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxpk2\" (UniqueName: \"kubernetes.io/projected/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-api-access-zxpk2\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.898860 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.898828    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6515eee-e003-4a20-a44e-1d514def0ddb-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.899137 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.899117    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.899218 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.899184    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.900909 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.900886    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6515eee-e003-4a20-a44e-1d514def0ddb-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:01.906953 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:01.906924    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxpk2\" (UniqueName: \"kubernetes.io/projected/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-api-access-zxpk2\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jtpt6\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:02.078771 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:02.078686    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:02.352686 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:02.352659    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"]
Sep 05 03:04:02.355207 ip-10-0-135-239 kubenswrapper[2555]: W0905 03:04:02.355178    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6515eee_e003_4a20_a44e_1d514def0ddb.slice/crio-65ad4a9ae7dba529813012d82484eba5230a98caf98abde050b3873fad8cca04 WatchSource:0}: Error finding container 65ad4a9ae7dba529813012d82484eba5230a98caf98abde050b3873fad8cca04: Status 404 returned error can't find the container with id 65ad4a9ae7dba529813012d82484eba5230a98caf98abde050b3873fad8cca04
Sep 05 03:04:02.484494 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:02.484456    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" event={"ID":"c6515eee-e003-4a20-a44e-1d514def0ddb","Type":"ContainerStarted","Data":"92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785"}
Sep 05 03:04:02.484707 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:02.484515    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" event={"ID":"c6515eee-e003-4a20-a44e-1d514def0ddb","Type":"ContainerStarted","Data":"65ad4a9ae7dba529813012d82484eba5230a98caf98abde050b3873fad8cca04"}
Sep 05 03:04:07.508283 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:07.508246    2555 generic.go:356] "Generic (PLEG): container finished" podID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerID="92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785" exitCode=0
Sep 05 03:04:07.508735 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:07.508319    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" event={"ID":"c6515eee-e003-4a20-a44e-1d514def0ddb","Type":"ContainerDied","Data":"92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785"}
Sep 05 03:04:18.569477 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:18.569439    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" event={"ID":"c6515eee-e003-4a20-a44e-1d514def0ddb","Type":"ContainerStarted","Data":"8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12"}
Sep 05 03:04:18.569886 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:18.569507    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" event={"ID":"c6515eee-e003-4a20-a44e-1d514def0ddb","Type":"ContainerStarted","Data":"215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b"}
Sep 05 03:04:18.569886 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:18.569723    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:18.590582 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:18.590517    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" podStartSLOduration=2.447978 podStartE2EDuration="17.590500337s" totalImagesPullingTime="10.612318761s" totalInitContainerRuntime="4.530203576s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:04:18.589439295 +0000 UTC m=+2779.597935453" watchObservedRunningTime="2026-09-05 03:04:18.590500337 +0000 UTC m=+2779.598996476"
Sep 05 03:04:19.573462 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:19.573430    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:19.574912 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:19.574883    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 05 03:04:20.578748 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:20.578710    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 05 03:04:25.583046 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:25.582974    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:04:25.583499 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:25.583468    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 05 03:04:35.583711 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:35.583669    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 05 03:04:45.583665 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:45.583620    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 05 03:04:55.583863 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:04:55.583820    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 05 03:05:05.584421 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:05.584388    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:05:12.883912 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:12.883876    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"]
Sep 05 03:05:12.884304 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:12.884155    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kserve-container" containerID="cri-o://215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b" gracePeriod=30
Sep 05 03:05:12.884304 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:12.884246    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kube-rbac-proxy" containerID="cri-o://8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12" gracePeriod=30
Sep 05 03:05:13.815373 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:13.815341    2555 generic.go:356] "Generic (PLEG): container finished" podID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerID="8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12" exitCode=2
Sep 05 03:05:13.815591 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:13.815410    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" event={"ID":"c6515eee-e003-4a20-a44e-1d514def0ddb","Type":"ContainerDied","Data":"8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12"}
Sep 05 03:05:15.579261 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:15.579218    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 05 03:05:15.583663 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:15.583634    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 05 03:05:18.839027 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.838997    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:05:18.839924 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.839901    2555 generic.go:356] "Generic (PLEG): container finished" podID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerID="215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b" exitCode=0
Sep 05 03:05:18.840010 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.839978    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" event={"ID":"c6515eee-e003-4a20-a44e-1d514def0ddb","Type":"ContainerDied","Data":"215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b"}
Sep 05 03:05:18.840077 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.840062    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6" event={"ID":"c6515eee-e003-4a20-a44e-1d514def0ddb","Type":"ContainerDied","Data":"65ad4a9ae7dba529813012d82484eba5230a98caf98abde050b3873fad8cca04"}
Sep 05 03:05:18.840135 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.840114    2555 scope.go:117] "RemoveContainer" containerID="8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12"
Sep 05 03:05:18.848356 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.848324    2555 scope.go:117] "RemoveContainer" containerID="215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b"
Sep 05 03:05:18.855842 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.855817    2555 scope.go:117] "RemoveContainer" containerID="92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785"
Sep 05 03:05:18.864371 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.864353    2555 scope.go:117] "RemoveContainer" containerID="8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12"
Sep 05 03:05:18.864770 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:05:18.864747    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12\": container with ID starting with 8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12 not found: ID does not exist" containerID="8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12"
Sep 05 03:05:18.864846 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.864779    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12"} err="failed to get container status \"8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12\": rpc error: code = NotFound desc = could not find container \"8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12\": container with ID starting with 8fab19138e02896aa0a4766cb23d632293a29952f1554b1bd432ec040a615f12 not found: ID does not exist"
Sep 05 03:05:18.864846 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.864799    2555 scope.go:117] "RemoveContainer" containerID="215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b"
Sep 05 03:05:18.865054 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:05:18.865040    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b\": container with ID starting with 215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b not found: ID does not exist" containerID="215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b"
Sep 05 03:05:18.865098 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.865058    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b"} err="failed to get container status \"215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b\": rpc error: code = NotFound desc = could not find container \"215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b\": container with ID starting with 215c24a1ca8e367851c3937f76937f8a0d142570a79ed598cb655dd69c82aa1b not found: ID does not exist"
Sep 05 03:05:18.865098 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.865072    2555 scope.go:117] "RemoveContainer" containerID="92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785"
Sep 05 03:05:18.865407 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:05:18.865388    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785\": container with ID starting with 92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785 not found: ID does not exist" containerID="92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785"
Sep 05 03:05:18.865461 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.865416    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785"} err="failed to get container status \"92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785\": rpc error: code = NotFound desc = could not find container \"92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785\": container with ID starting with 92114fda5c433cf8c593441f29f4e0a5104e940843c2acd7ba2b1249e28ff785 not found: ID does not exist"
Sep 05 03:05:18.956754 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.956664    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxpk2\" (UniqueName: \"kubernetes.io/projected/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-api-access-zxpk2\") pod \"c6515eee-e003-4a20-a44e-1d514def0ddb\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") "
Sep 05 03:05:18.956754 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.956708    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-rbac-proxy-sar-config\") pod \"c6515eee-e003-4a20-a44e-1d514def0ddb\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") "
Sep 05 03:05:18.956754 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.956742    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6515eee-e003-4a20-a44e-1d514def0ddb-proxy-tls\") pod \"c6515eee-e003-4a20-a44e-1d514def0ddb\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") "
Sep 05 03:05:18.956987 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.956800    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-cabundle-cert\") pod \"c6515eee-e003-4a20-a44e-1d514def0ddb\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") "
Sep 05 03:05:18.956987 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.956827    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6515eee-e003-4a20-a44e-1d514def0ddb-kserve-provision-location\") pod \"c6515eee-e003-4a20-a44e-1d514def0ddb\" (UID: \"c6515eee-e003-4a20-a44e-1d514def0ddb\") "
Sep 05 03:05:18.957221 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.957189    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6515eee-e003-4a20-a44e-1d514def0ddb" (UID: "c6515eee-e003-4a20-a44e-1d514def0ddb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:05:18.957221 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.957208    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6515eee-e003-4a20-a44e-1d514def0ddb" (UID: "c6515eee-e003-4a20-a44e-1d514def0ddb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:05:18.958868 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.958834    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-api-access-zxpk2" (OuterVolumeSpecName: "kube-api-access-zxpk2") pod "c6515eee-e003-4a20-a44e-1d514def0ddb" (UID: "c6515eee-e003-4a20-a44e-1d514def0ddb"). InnerVolumeSpecName "kube-api-access-zxpk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:05:18.958868 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.958849    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6515eee-e003-4a20-a44e-1d514def0ddb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6515eee-e003-4a20-a44e-1d514def0ddb" (UID: "c6515eee-e003-4a20-a44e-1d514def0ddb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:05:18.969142 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:18.969106    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6515eee-e003-4a20-a44e-1d514def0ddb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6515eee-e003-4a20-a44e-1d514def0ddb" (UID: "c6515eee-e003-4a20-a44e-1d514def0ddb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:05:19.057403 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:19.057349    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:05:19.057403 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:19.057395    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6515eee-e003-4a20-a44e-1d514def0ddb-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:05:19.057403 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:19.057405    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxpk2\" (UniqueName: \"kubernetes.io/projected/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-api-access-zxpk2\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:05:19.057403 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:19.057414    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6515eee-e003-4a20-a44e-1d514def0ddb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:05:19.057403 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:19.057424    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6515eee-e003-4a20-a44e-1d514def0ddb-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:05:19.845791 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:19.845710    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"
Sep 05 03:05:19.863804 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:19.863767    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"]
Sep 05 03:05:19.868009 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:19.867979    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jtpt6"]
Sep 05 03:05:21.544302 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:05:21.544261    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" path="/var/lib/kubelet/pods/c6515eee-e003-4a20-a44e-1d514def0ddb/volumes"
Sep 05 03:06:34.676973 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.676936    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"]
Sep 05 03:06:34.677623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.677264    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="storage-initializer"
Sep 05 03:06:34.677623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.677275    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="storage-initializer"
Sep 05 03:06:34.677623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.677290    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kserve-container"
Sep 05 03:06:34.677623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.677296    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kserve-container"
Sep 05 03:06:34.677623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.677309    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kube-rbac-proxy"
Sep 05 03:06:34.677623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.677314    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kube-rbac-proxy"
Sep 05 03:06:34.677623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.677361    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kube-rbac-proxy"
Sep 05 03:06:34.677623 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.677370    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6515eee-e003-4a20-a44e-1d514def0ddb" containerName="kserve-container"
Sep 05 03:06:34.680071 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.680052    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:34.682519 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.682501    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 05 03:06:34.682769 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.682516    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 03:06:34.682926 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.682906    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 03:06:34.683453 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.683432    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 05 03:06:34.683511 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.683464    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 03:06:34.683511 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.683504    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 05 03:06:34.691249 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.691229    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"]
Sep 05 03:06:34.786964 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.786933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b2655df-d02d-4444-8acf-f8c752191259-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:34.787141 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.786985    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjxtl\" (UniqueName: \"kubernetes.io/projected/1b2655df-d02d-4444-8acf-f8c752191259-kube-api-access-pjxtl\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:34.787141 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.787006    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b2655df-d02d-4444-8acf-f8c752191259-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:34.787141 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.787057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b2655df-d02d-4444-8acf-f8c752191259-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:34.888026 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.887990    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxtl\" (UniqueName: \"kubernetes.io/projected/1b2655df-d02d-4444-8acf-f8c752191259-kube-api-access-pjxtl\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:34.888175 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.888030    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b2655df-d02d-4444-8acf-f8c752191259-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:34.888175 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.888068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b2655df-d02d-4444-8acf-f8c752191259-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:34.888175 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.888102    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b2655df-d02d-4444-8acf-f8c752191259-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:34.888325 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:06:34.888200    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 05 03:06:34.888325 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:06:34.888271    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b2655df-d02d-4444-8acf-f8c752191259-proxy-tls podName:1b2655df-d02d-4444-8acf-f8c752191259 nodeName:}" failed. No retries permitted until 2026-09-05 03:06:35.388252394 +0000 UTC m=+2916.396748512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1b2655df-d02d-4444-8acf-f8c752191259-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w" (UID: "1b2655df-d02d-4444-8acf-f8c752191259") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 05 03:06:34.888569 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.888517    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b2655df-d02d-4444-8acf-f8c752191259-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:34.888837 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.888820    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b2655df-d02d-4444-8acf-f8c752191259-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:34.898172 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:34.898151    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxtl\" (UniqueName: \"kubernetes.io/projected/1b2655df-d02d-4444-8acf-f8c752191259-kube-api-access-pjxtl\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:35.392836 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:35.392794    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b2655df-d02d-4444-8acf-f8c752191259-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:35.395201 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:35.395181    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b2655df-d02d-4444-8acf-f8c752191259-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:35.590987 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:35.590934    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:35.871195 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:35.871169    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"]
Sep 05 03:06:35.873449 ip-10-0-135-239 kubenswrapper[2555]: W0905 03:06:35.873419    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2655df_d02d_4444_8acf_f8c752191259.slice/crio-72c1e3174f17a95d01aee01a618bd5fd16321c3f26d1d2e1126c404a8e20f415 WatchSource:0}: Error finding container 72c1e3174f17a95d01aee01a618bd5fd16321c3f26d1d2e1126c404a8e20f415: Status 404 returned error can't find the container with id 72c1e3174f17a95d01aee01a618bd5fd16321c3f26d1d2e1126c404a8e20f415
Sep 05 03:06:36.181816 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:36.181775    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w" event={"ID":"1b2655df-d02d-4444-8acf-f8c752191259","Type":"ContainerStarted","Data":"a21ba5ab690cde0d48b12c40c08834ea5e73443948fc12c7bf327364504fb747"}
Sep 05 03:06:36.181996 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:36.181845    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w" event={"ID":"1b2655df-d02d-4444-8acf-f8c752191259","Type":"ContainerStarted","Data":"72c1e3174f17a95d01aee01a618bd5fd16321c3f26d1d2e1126c404a8e20f415"}
Sep 05 03:06:39.196643 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:39.196613    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w_1b2655df-d02d-4444-8acf-f8c752191259/storage-initializer/0.log"
Sep 05 03:06:39.197066 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:39.196650    2555 generic.go:356] "Generic (PLEG): container finished" podID="1b2655df-d02d-4444-8acf-f8c752191259" containerID="a21ba5ab690cde0d48b12c40c08834ea5e73443948fc12c7bf327364504fb747" exitCode=1
Sep 05 03:06:39.197066 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:39.196725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w" event={"ID":"1b2655df-d02d-4444-8acf-f8c752191259","Type":"ContainerDied","Data":"a21ba5ab690cde0d48b12c40c08834ea5e73443948fc12c7bf327364504fb747"}
Sep 05 03:06:40.202936 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:40.202908    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w_1b2655df-d02d-4444-8acf-f8c752191259/storage-initializer/0.log"
Sep 05 03:06:40.203399 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:40.202972    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w" event={"ID":"1b2655df-d02d-4444-8acf-f8c752191259","Type":"ContainerStarted","Data":"a6421d2795b69eea2ac8a8ab2a2be7801f09360b1f1247ccfa9279a4fc6a8b81"}
Sep 05 03:06:42.213201 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:42.213173    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w_1b2655df-d02d-4444-8acf-f8c752191259/storage-initializer/1.log"
Sep 05 03:06:42.213627 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:42.213608    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w_1b2655df-d02d-4444-8acf-f8c752191259/storage-initializer/0.log"
Sep 05 03:06:42.213670 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:42.213642    2555 generic.go:356] "Generic (PLEG): container finished" podID="1b2655df-d02d-4444-8acf-f8c752191259" containerID="a6421d2795b69eea2ac8a8ab2a2be7801f09360b1f1247ccfa9279a4fc6a8b81" exitCode=1
Sep 05 03:06:42.213718 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:42.213704    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w" event={"ID":"1b2655df-d02d-4444-8acf-f8c752191259","Type":"ContainerDied","Data":"a6421d2795b69eea2ac8a8ab2a2be7801f09360b1f1247ccfa9279a4fc6a8b81"}
Sep 05 03:06:42.213773 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:42.213762    2555 scope.go:117] "RemoveContainer" containerID="a21ba5ab690cde0d48b12c40c08834ea5e73443948fc12c7bf327364504fb747"
Sep 05 03:06:42.214107 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:42.214087    2555 scope.go:117] "RemoveContainer" containerID="a21ba5ab690cde0d48b12c40c08834ea5e73443948fc12c7bf327364504fb747"
Sep 05 03:06:42.224474 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:06:42.224448    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w_kserve-ci-e2e-test(1b2655df-d02d-4444-8acf-f8c752191259)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w" podUID="1b2655df-d02d-4444-8acf-f8c752191259"
Sep 05 03:06:42.224610 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:06:42.224583    2555 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w_kserve-ci-e2e-test_1b2655df-d02d-4444-8acf-f8c752191259_0 in pod sandbox 72c1e3174f17a95d01aee01a618bd5fd16321c3f26d1d2e1126c404a8e20f415 from index: no such id: 'a21ba5ab690cde0d48b12c40c08834ea5e73443948fc12c7bf327364504fb747'" containerID="a21ba5ab690cde0d48b12c40c08834ea5e73443948fc12c7bf327364504fb747"
Sep 05 03:06:42.224668 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:42.224621    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21ba5ab690cde0d48b12c40c08834ea5e73443948fc12c7bf327364504fb747"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w_kserve-ci-e2e-test_1b2655df-d02d-4444-8acf-f8c752191259_0 in pod sandbox 72c1e3174f17a95d01aee01a618bd5fd16321c3f26d1d2e1126c404a8e20f415 from index: no such id: 'a21ba5ab690cde0d48b12c40c08834ea5e73443948fc12c7bf327364504fb747'"
Sep 05 03:06:43.219803 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:43.219776    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w_1b2655df-d02d-4444-8acf-f8c752191259/storage-initializer/1.log"
Sep 05 03:06:44.629849 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.629804    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"]
Sep 05 03:06:44.762557 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.762521    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w_1b2655df-d02d-4444-8acf-f8c752191259/storage-initializer/1.log"
Sep 05 03:06:44.765983 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.765967    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:44.776363 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.776340    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b2655df-d02d-4444-8acf-f8c752191259-kube-rbac-proxy-sar-config\") pod \"1b2655df-d02d-4444-8acf-f8c752191259\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") "
Sep 05 03:06:44.776485 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.776395    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjxtl\" (UniqueName: \"kubernetes.io/projected/1b2655df-d02d-4444-8acf-f8c752191259-kube-api-access-pjxtl\") pod \"1b2655df-d02d-4444-8acf-f8c752191259\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") "
Sep 05 03:06:44.776485 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.776473    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b2655df-d02d-4444-8acf-f8c752191259-proxy-tls\") pod \"1b2655df-d02d-4444-8acf-f8c752191259\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") "
Sep 05 03:06:44.776632 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.776544    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b2655df-d02d-4444-8acf-f8c752191259-kserve-provision-location\") pod \"1b2655df-d02d-4444-8acf-f8c752191259\" (UID: \"1b2655df-d02d-4444-8acf-f8c752191259\") "
Sep 05 03:06:44.776767 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.776739    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2655df-d02d-4444-8acf-f8c752191259-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b2655df-d02d-4444-8acf-f8c752191259" (UID: "1b2655df-d02d-4444-8acf-f8c752191259"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:06:44.776875 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.776847    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2655df-d02d-4444-8acf-f8c752191259-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b2655df-d02d-4444-8acf-f8c752191259" (UID: "1b2655df-d02d-4444-8acf-f8c752191259"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:06:44.778444 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.778417    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b2655df-d02d-4444-8acf-f8c752191259-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b2655df-d02d-4444-8acf-f8c752191259" (UID: "1b2655df-d02d-4444-8acf-f8c752191259"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:06:44.778569 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.778546    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2655df-d02d-4444-8acf-f8c752191259-kube-api-access-pjxtl" (OuterVolumeSpecName: "kube-api-access-pjxtl") pod "1b2655df-d02d-4444-8acf-f8c752191259" (UID: "1b2655df-d02d-4444-8acf-f8c752191259"). InnerVolumeSpecName "kube-api-access-pjxtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:06:44.877684 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.877652    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b2655df-d02d-4444-8acf-f8c752191259-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:06:44.877684 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.877682    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b2655df-d02d-4444-8acf-f8c752191259-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:06:44.877684 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.877691    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjxtl\" (UniqueName: \"kubernetes.io/projected/1b2655df-d02d-4444-8acf-f8c752191259-kube-api-access-pjxtl\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:06:44.877928 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:44.877700    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b2655df-d02d-4444-8acf-f8c752191259-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:06:45.228998 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:45.228971    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w_1b2655df-d02d-4444-8acf-f8c752191259/storage-initializer/1.log"
Sep 05 03:06:45.229199 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:45.229077    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"
Sep 05 03:06:45.229199 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:45.229102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w" event={"ID":"1b2655df-d02d-4444-8acf-f8c752191259","Type":"ContainerDied","Data":"72c1e3174f17a95d01aee01a618bd5fd16321c3f26d1d2e1126c404a8e20f415"}
Sep 05 03:06:45.229199 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:45.229160    2555 scope.go:117] "RemoveContainer" containerID="a6421d2795b69eea2ac8a8ab2a2be7801f09360b1f1247ccfa9279a4fc6a8b81"
Sep 05 03:06:45.270546 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:45.270497    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"]
Sep 05 03:06:45.275473 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:45.275448    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-85dbf98698-mhv8w"]
Sep 05 03:06:45.543622 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:06:45.543528    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b2655df-d02d-4444-8acf-f8c752191259" path="/var/lib/kubelet/pods/1b2655df-d02d-4444-8acf-f8c752191259/volumes"
Sep 05 03:07:59.707066 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:07:59.707036    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 03:07:59.710187 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:07:59.710166    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 03:08:10.853503 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.853465    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"]
Sep 05 03:08:10.853945 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.853805    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b2655df-d02d-4444-8acf-f8c752191259" containerName="storage-initializer"
Sep 05 03:08:10.853945 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.853817    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b2655df-d02d-4444-8acf-f8c752191259" containerName="storage-initializer"
Sep 05 03:08:10.853945 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.853881    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b2655df-d02d-4444-8acf-f8c752191259" containerName="storage-initializer"
Sep 05 03:08:10.854046 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.853949    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b2655df-d02d-4444-8acf-f8c752191259" containerName="storage-initializer"
Sep 05 03:08:10.854046 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.853955    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b2655df-d02d-4444-8acf-f8c752191259" containerName="storage-initializer"
Sep 05 03:08:10.854046 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.854013    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b2655df-d02d-4444-8acf-f8c752191259" containerName="storage-initializer"
Sep 05 03:08:10.857180 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.857159    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:10.859662 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.859642    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 05 03:08:10.859759 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.859666    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 03:08:10.860002 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.859986    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 03:08:10.860527 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.860510    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 05 03:08:10.860652 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.860590    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:08:10.860652 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.860599    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 03:08:10.860652 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.860603    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 03:08:10.873003 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.872981    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"]
Sep 05 03:08:10.997178 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.997143    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:10.997354 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.997191    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:10.997354 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.997221    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:10.997354 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.997260    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:10.997354 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:10.997300    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krtdt\" (UniqueName: \"kubernetes.io/projected/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-api-access-krtdt\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.098128 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.098090    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.098128 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.098132    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.098394 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.098150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.098394 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.098168    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krtdt\" (UniqueName: \"kubernetes.io/projected/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-api-access-krtdt\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.098394 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.098220    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.098637 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.098614    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.098924 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.098907    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.098976 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.098965    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.100710 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.100692    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.106991 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.106939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krtdt\" (UniqueName: \"kubernetes.io/projected/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-api-access-krtdt\") pod \"isvc-pmml-runtime-predictor-885fc8777-h5vbx\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.167883 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.167829    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:11.439319 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.439292    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"]
Sep 05 03:08:11.441556 ip-10-0-135-239 kubenswrapper[2555]: W0905 03:08:11.441510    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e2f35b_0a31_4d5c_a50d_62ff08cd7a2f.slice/crio-dadb578949cd339cb26bf690abc60570161b1809dbed27d868f2606778d541ad WatchSource:0}: Error finding container dadb578949cd339cb26bf690abc60570161b1809dbed27d868f2606778d541ad: Status 404 returned error can't find the container with id dadb578949cd339cb26bf690abc60570161b1809dbed27d868f2606778d541ad
Sep 05 03:08:11.601170 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.601130    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" event={"ID":"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f","Type":"ContainerStarted","Data":"5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a"}
Sep 05 03:08:11.601366 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:11.601190    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" event={"ID":"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f","Type":"ContainerStarted","Data":"dadb578949cd339cb26bf690abc60570161b1809dbed27d868f2606778d541ad"}
Sep 05 03:08:15.620997 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:15.620961    2555 generic.go:356] "Generic (PLEG): container finished" podID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerID="5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a" exitCode=0
Sep 05 03:08:15.621374 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:15.621014    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" event={"ID":"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f","Type":"ContainerDied","Data":"5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a"}
Sep 05 03:08:15.622240 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:15.622223    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 03:08:16.900866 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:16.898935    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"]
Sep 05 03:08:16.906972 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:16.906704    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:16.914220 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:16.913869    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"]
Sep 05 03:08:16.914220 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:16.914051    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 05 03:08:16.914220 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:16.914095    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 05 03:08:16.914220 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:16.914050    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 05 03:08:17.050515 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.050429    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2wn8\" (UniqueName: \"kubernetes.io/projected/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-api-access-p2wn8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.050723 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.050562    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d85d8e1-b60c-489a-b876-24f872e4cd60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.050723 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.050611    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d85d8e1-b60c-489a-b876-24f872e4cd60-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.050723 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.050681    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.151853 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.151225    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2wn8\" (UniqueName: \"kubernetes.io/projected/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-api-access-p2wn8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.151853 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.151312    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d85d8e1-b60c-489a-b876-24f872e4cd60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.151853 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.151350    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d85d8e1-b60c-489a-b876-24f872e4cd60-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.151853 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.151409    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.152729 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.152206    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d85d8e1-b60c-489a-b876-24f872e4cd60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.152729 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.152671    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.155465 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.155421    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d85d8e1-b60c-489a-b876-24f872e4cd60-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.168834 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.168807    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2wn8\" (UniqueName: \"kubernetes.io/projected/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-api-access-p2wn8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:17.226478 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:17.226440    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:18.501227 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:18.501030    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"]
Sep 05 03:08:19.869551 ip-10-0-135-239 kubenswrapper[2555]: W0905 03:08:19.869497    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d85d8e1_b60c_489a_b876_24f872e4cd60.slice/crio-8301f3c18a3bdbbc71505a89d9b26f75a87daca9a9f1a850f5aba67ad69c349a WatchSource:0}: Error finding container 8301f3c18a3bdbbc71505a89d9b26f75a87daca9a9f1a850f5aba67ad69c349a: Status 404 returned error can't find the container with id 8301f3c18a3bdbbc71505a89d9b26f75a87daca9a9f1a850f5aba67ad69c349a
Sep 05 03:08:20.654446 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:20.654407    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97" event={"ID":"9d85d8e1-b60c-489a-b876-24f872e4cd60","Type":"ContainerStarted","Data":"8301f3c18a3bdbbc71505a89d9b26f75a87daca9a9f1a850f5aba67ad69c349a"}
Sep 05 03:08:22.666235 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:22.666208    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97" event={"ID":"9d85d8e1-b60c-489a-b876-24f872e4cd60","Type":"ContainerStarted","Data":"f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af"}
Sep 05 03:08:23.673024 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:23.672987    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" event={"ID":"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f","Type":"ContainerStarted","Data":"7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716"}
Sep 05 03:08:23.673415 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:23.673049    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" event={"ID":"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f","Type":"ContainerStarted","Data":"d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0"}
Sep 05 03:08:23.673415 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:23.673311    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:23.694396 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:23.694346    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podStartSLOduration=2.7473433050000002 podStartE2EDuration="13.694327264s" totalImagesPullingTime="7.032962664s" totalInitContainerRuntime="3.914021295s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:08:23.693917771 +0000 UTC m=+3024.702413925" watchObservedRunningTime="2026-09-05 03:08:23.694327264 +0000 UTC m=+3024.702823416"
Sep 05 03:08:24.678447 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:24.678412    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:24.679807 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:24.679780    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 05 03:08:25.684738 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:25.684708    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97_9d85d8e1-b60c-489a-b876-24f872e4cd60/storage-initializer/0.log"
Sep 05 03:08:25.685127 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:25.684749    2555 generic.go:356] "Generic (PLEG): container finished" podID="9d85d8e1-b60c-489a-b876-24f872e4cd60" containerID="f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af" exitCode=1
Sep 05 03:08:25.685127 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:25.684830    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97" event={"ID":"9d85d8e1-b60c-489a-b876-24f872e4cd60","Type":"ContainerDied","Data":"f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af"}
Sep 05 03:08:25.685216 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:25.685174    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 05 03:08:26.694147 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:26.694098    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97_9d85d8e1-b60c-489a-b876-24f872e4cd60/storage-initializer/0.log"
Sep 05 03:08:26.694571 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:26.694170    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97" event={"ID":"9d85d8e1-b60c-489a-b876-24f872e4cd60","Type":"ContainerStarted","Data":"6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8"}
Sep 05 03:08:26.877479 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:26.877443    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"]
Sep 05 03:08:27.698962 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:27.698894    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97" podUID="9d85d8e1-b60c-489a-b876-24f872e4cd60" containerName="storage-initializer" containerID="cri-o://6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8" gracePeriod=30
Sep 05 03:08:29.646698 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.646675    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97_9d85d8e1-b60c-489a-b876-24f872e4cd60/storage-initializer/1.log"
Sep 05 03:08:29.647071 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.647058    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97_9d85d8e1-b60c-489a-b876-24f872e4cd60/storage-initializer/0.log"
Sep 05 03:08:29.649367 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.649353    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:29.708991 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.708964    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97_9d85d8e1-b60c-489a-b876-24f872e4cd60/storage-initializer/1.log"
Sep 05 03:08:29.709356 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.709342    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97_9d85d8e1-b60c-489a-b876-24f872e4cd60/storage-initializer/0.log"
Sep 05 03:08:29.709439 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.709374    2555 generic.go:356] "Generic (PLEG): container finished" podID="9d85d8e1-b60c-489a-b876-24f872e4cd60" containerID="6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8" exitCode=1
Sep 05 03:08:29.709498 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.709455    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"
Sep 05 03:08:29.709498 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.709465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97" event={"ID":"9d85d8e1-b60c-489a-b876-24f872e4cd60","Type":"ContainerDied","Data":"6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8"}
Sep 05 03:08:29.709637 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.709522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97" event={"ID":"9d85d8e1-b60c-489a-b876-24f872e4cd60","Type":"ContainerDied","Data":"8301f3c18a3bdbbc71505a89d9b26f75a87daca9a9f1a850f5aba67ad69c349a"}
Sep 05 03:08:29.709637 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.709569    2555 scope.go:117] "RemoveContainer" containerID="6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8"
Sep 05 03:08:29.717489 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.717473    2555 scope.go:117] "RemoveContainer" containerID="f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af"
Sep 05 03:08:29.724883 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.724864    2555 scope.go:117] "RemoveContainer" containerID="6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8"
Sep 05 03:08:29.725141 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:08:29.725123    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8\": container with ID starting with 6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8 not found: ID does not exist" containerID="6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8"
Sep 05 03:08:29.725190 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.725150    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8"} err="failed to get container status \"6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8\": rpc error: code = NotFound desc = could not find container \"6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8\": container with ID starting with 6c2b9ca09576047d0c35af27010172949d1ca80cc12f22ce7f5e79fc8ac3edb8 not found: ID does not exist"
Sep 05 03:08:29.725190 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.725169    2555 scope.go:117] "RemoveContainer" containerID="f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af"
Sep 05 03:08:29.725381 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:08:29.725368    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af\": container with ID starting with f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af not found: ID does not exist" containerID="f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af"
Sep 05 03:08:29.725422 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.725383    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af"} err="failed to get container status \"f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af\": rpc error: code = NotFound desc = could not find container \"f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af\": container with ID starting with f82680ee1fe06f6aab751a5020f8e46a3f951feac6d8a6a80ad1bfc3fd46a6af not found: ID does not exist"
Sep 05 03:08:29.764811 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.764730    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-rbac-proxy-sar-config\") pod \"9d85d8e1-b60c-489a-b876-24f872e4cd60\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") "
Sep 05 03:08:29.764811 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.764789    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2wn8\" (UniqueName: \"kubernetes.io/projected/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-api-access-p2wn8\") pod \"9d85d8e1-b60c-489a-b876-24f872e4cd60\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") "
Sep 05 03:08:29.764997 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.764820    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d85d8e1-b60c-489a-b876-24f872e4cd60-proxy-tls\") pod \"9d85d8e1-b60c-489a-b876-24f872e4cd60\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") "
Sep 05 03:08:29.764997 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.764985    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d85d8e1-b60c-489a-b876-24f872e4cd60-kserve-provision-location\") pod \"9d85d8e1-b60c-489a-b876-24f872e4cd60\" (UID: \"9d85d8e1-b60c-489a-b876-24f872e4cd60\") "
Sep 05 03:08:29.765228 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.765205    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d85d8e1-b60c-489a-b876-24f872e4cd60" (UID: "9d85d8e1-b60c-489a-b876-24f872e4cd60"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:08:29.765228 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.765216    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d85d8e1-b60c-489a-b876-24f872e4cd60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d85d8e1-b60c-489a-b876-24f872e4cd60" (UID: "9d85d8e1-b60c-489a-b876-24f872e4cd60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:08:29.765331 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.765312    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d85d8e1-b60c-489a-b876-24f872e4cd60-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:08:29.765331 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.765326    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:08:29.766878 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.766844    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d85d8e1-b60c-489a-b876-24f872e4cd60-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d85d8e1-b60c-489a-b876-24f872e4cd60" (UID: "9d85d8e1-b60c-489a-b876-24f872e4cd60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:08:29.766964 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.766914    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-api-access-p2wn8" (OuterVolumeSpecName: "kube-api-access-p2wn8") pod "9d85d8e1-b60c-489a-b876-24f872e4cd60" (UID: "9d85d8e1-b60c-489a-b876-24f872e4cd60"). InnerVolumeSpecName "kube-api-access-p2wn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:08:29.866624 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.866584    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2wn8\" (UniqueName: \"kubernetes.io/projected/9d85d8e1-b60c-489a-b876-24f872e4cd60-kube-api-access-p2wn8\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:08:29.866624 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:29.866617    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d85d8e1-b60c-489a-b876-24f872e4cd60-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:08:30.049652 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:30.049615    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"]
Sep 05 03:08:30.054628 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:30.054603    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-54c8f866dc-6wl97"]
Sep 05 03:08:30.690024 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:30.689996    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:08:30.690524 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:30.690493    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 05 03:08:31.544502 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:31.544468    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d85d8e1-b60c-489a-b876-24f872e4cd60" path="/var/lib/kubelet/pods/9d85d8e1-b60c-489a-b876-24f872e4cd60/volumes"
Sep 05 03:08:40.691274 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:40.691232    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 05 03:08:50.691030 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:08:50.690990    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 05 03:09:00.691333 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:00.691291    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 05 03:09:10.690672 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:10.690634    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 05 03:09:20.690892 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:20.690850    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 05 03:09:30.691307 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:30.691262    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 05 03:09:40.692301 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:40.692269    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:09:51.871626 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:51.871587    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"]
Sep 05 03:09:51.872143 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:51.871846    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container" containerID="cri-o://d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0" gracePeriod=30
Sep 05 03:09:51.872143 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:51.871888    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kube-rbac-proxy" containerID="cri-o://7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716" gracePeriod=30
Sep 05 03:09:52.078027 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:52.077971    2555 generic.go:356] "Generic (PLEG): container finished" podID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerID="7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716" exitCode=2
Sep 05 03:09:52.078199 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:52.078048    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" event={"ID":"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f","Type":"ContainerDied","Data":"7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716"}
Sep 05 03:09:55.524179 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.524152    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:09:55.575971 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.575942    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krtdt\" (UniqueName: \"kubernetes.io/projected/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-api-access-krtdt\") pod \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") "
Sep 05 03:09:55.576179 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.575983    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-rbac-proxy-sar-config\") pod \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") "
Sep 05 03:09:55.576179 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.576025    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-proxy-tls\") pod \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") "
Sep 05 03:09:55.576179 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.576055    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-cabundle-cert\") pod \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") "
Sep 05 03:09:55.576179 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.576102    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kserve-provision-location\") pod \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\" (UID: \"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f\") "
Sep 05 03:09:55.576464 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.576434    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" (UID: "75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:09:55.576518 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.576434    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" (UID: "75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:09:55.576518 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.576442    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" (UID: "75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:09:55.578651 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.578617    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" (UID: "75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:09:55.578758 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.578653    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-api-access-krtdt" (OuterVolumeSpecName: "kube-api-access-krtdt") pod "75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" (UID: "75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f"). InnerVolumeSpecName "kube-api-access-krtdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:09:55.677666 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.677628    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:09:55.677666 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.677660    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krtdt\" (UniqueName: \"kubernetes.io/projected/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-api-access-krtdt\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:09:55.677666 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.677671    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:09:55.677900 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.677680    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:09:55.677900 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:55.677691    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:09:56.098074 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.097985    2555 generic.go:356] "Generic (PLEG): container finished" podID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerID="d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0" exitCode=0
Sep 05 03:09:56.098074 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.098054    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"
Sep 05 03:09:56.098074 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.098063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" event={"ID":"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f","Type":"ContainerDied","Data":"d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0"}
Sep 05 03:09:56.098303 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.098134    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx" event={"ID":"75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f","Type":"ContainerDied","Data":"dadb578949cd339cb26bf690abc60570161b1809dbed27d868f2606778d541ad"}
Sep 05 03:09:56.098303 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.098176    2555 scope.go:117] "RemoveContainer" containerID="7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716"
Sep 05 03:09:56.106734 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.106708    2555 scope.go:117] "RemoveContainer" containerID="d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0"
Sep 05 03:09:56.114299 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.114282    2555 scope.go:117] "RemoveContainer" containerID="5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a"
Sep 05 03:09:56.123298 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.122800    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"]
Sep 05 03:09:56.124338 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.124321    2555 scope.go:117] "RemoveContainer" containerID="7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716"
Sep 05 03:09:56.124641 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:09:56.124617    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716\": container with ID starting with 7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716 not found: ID does not exist" containerID="7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716"
Sep 05 03:09:56.124716 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.124649    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716"} err="failed to get container status \"7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716\": rpc error: code = NotFound desc = could not find container \"7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716\": container with ID starting with 7bab0024bd3ae7b00af8857e1f90a1491694e3fed67713ea7c5905a2228a6716 not found: ID does not exist"
Sep 05 03:09:56.124716 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.124670    2555 scope.go:117] "RemoveContainer" containerID="d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0"
Sep 05 03:09:56.124938 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:09:56.124919    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0\": container with ID starting with d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0 not found: ID does not exist" containerID="d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0"
Sep 05 03:09:56.125002 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.124958    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0"} err="failed to get container status \"d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0\": rpc error: code = NotFound desc = could not find container \"d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0\": container with ID starting with d6d2809eaca9c71fa23ef1e436ba3a40bd424657bd0ea965a81664d49d569ca0 not found: ID does not exist"
Sep 05 03:09:56.125002 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.124986    2555 scope.go:117] "RemoveContainer" containerID="5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a"
Sep 05 03:09:56.125232 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:09:56.125215    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a\": container with ID starting with 5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a not found: ID does not exist" containerID="5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a"
Sep 05 03:09:56.125289 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.125237    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a"} err="failed to get container status \"5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a\": rpc error: code = NotFound desc = could not find container \"5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a\": container with ID starting with 5c50a5a4134f8a8a0f1c0fe6b4113561a58bcef851271da9a0efb84ff4f3060a not found: ID does not exist"
Sep 05 03:09:56.128924 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:56.128902    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-h5vbx"]
Sep 05 03:09:57.046488 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046453    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"]
Sep 05 03:09:57.046913 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046792    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container"
Sep 05 03:09:57.046913 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046804    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container"
Sep 05 03:09:57.046913 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046813    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kube-rbac-proxy"
Sep 05 03:09:57.046913 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046819    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kube-rbac-proxy"
Sep 05 03:09:57.046913 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046830    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d85d8e1-b60c-489a-b876-24f872e4cd60" containerName="storage-initializer"
Sep 05 03:09:57.046913 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046836    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d85d8e1-b60c-489a-b876-24f872e4cd60" containerName="storage-initializer"
Sep 05 03:09:57.046913 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046848    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="storage-initializer"
Sep 05 03:09:57.046913 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046854    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="storage-initializer"
Sep 05 03:09:57.047194 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046920    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d85d8e1-b60c-489a-b876-24f872e4cd60" containerName="storage-initializer"
Sep 05 03:09:57.047194 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046929    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kserve-container"
Sep 05 03:09:57.047194 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046936    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" containerName="kube-rbac-proxy"
Sep 05 03:09:57.047194 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046984    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d85d8e1-b60c-489a-b876-24f872e4cd60" containerName="storage-initializer"
Sep 05 03:09:57.047194 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.046990    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d85d8e1-b60c-489a-b876-24f872e4cd60" containerName="storage-initializer"
Sep 05 03:09:57.047194 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.047045    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d85d8e1-b60c-489a-b876-24f872e4cd60" containerName="storage-initializer"
Sep 05 03:09:57.051342 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.051323    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.053727 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.053702    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 05 03:09:57.053843 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.053756    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 03:09:57.053843 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.053773    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 03:09:57.054003 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.053988    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 03:09:57.054770 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.054751    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 05 03:09:57.054880 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.054770    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:09:57.054880 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.054848    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpkjg\""
Sep 05 03:09:57.062235 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.062214    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"]
Sep 05 03:09:57.087750 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.087718    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.087880 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.087774    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.087880 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.087820    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68d0f19-09ee-4980-90e9-49629ccc6fc6-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.087963 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.087876    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.087963 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.087911    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcr8g\" (UniqueName: \"kubernetes.io/projected/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-api-access-bcr8g\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.188581 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.188529    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.188774 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.188600    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.188774 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.188622    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68d0f19-09ee-4980-90e9-49629ccc6fc6-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.188774 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.188640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.188774 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.188669    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcr8g\" (UniqueName: \"kubernetes.io/projected/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-api-access-bcr8g\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.188774 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:09:57.188767    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 05 03:09:57.189012 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:09:57.188835    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68d0f19-09ee-4980-90e9-49629ccc6fc6-proxy-tls podName:e68d0f19-09ee-4980-90e9-49629ccc6fc6 nodeName:}" failed. No retries permitted until 2026-09-05 03:09:57.68881379 +0000 UTC m=+3118.697309906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e68d0f19-09ee-4980-90e9-49629ccc6fc6-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" (UID: "e68d0f19-09ee-4980-90e9-49629ccc6fc6") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 05 03:09:57.189113 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.189095    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.189280 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.189263    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.189318 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.189265    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.197452 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.197430    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcr8g\" (UniqueName: \"kubernetes.io/projected/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-api-access-bcr8g\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.544751 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.544718    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f" path="/var/lib/kubelet/pods/75e2f35b-0a31-4d5c-a50d-62ff08cd7a2f/volumes"
Sep 05 03:09:57.692693 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.692656    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68d0f19-09ee-4980-90e9-49629ccc6fc6-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.695045 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.695023    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68d0f19-09ee-4980-90e9-49629ccc6fc6-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:57.961710 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:57.961671    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:09:58.239326 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:58.239299    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"]
Sep 05 03:09:58.241944 ip-10-0-135-239 kubenswrapper[2555]: W0905 03:09:58.241914    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68d0f19_09ee_4980_90e9_49629ccc6fc6.slice/crio-91429d4fa8bbb7ea2fe9ce5c3d4da6f553150a10739e7ccb36426e58811ba346 WatchSource:0}: Error finding container 91429d4fa8bbb7ea2fe9ce5c3d4da6f553150a10739e7ccb36426e58811ba346: Status 404 returned error can't find the container with id 91429d4fa8bbb7ea2fe9ce5c3d4da6f553150a10739e7ccb36426e58811ba346
Sep 05 03:09:59.112896 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:59.112858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" event={"ID":"e68d0f19-09ee-4980-90e9-49629ccc6fc6","Type":"ContainerStarted","Data":"dd7a449ac68569f2dd45ede2d616ece0a17aee68cf966caf57b4d3442be562da"}
Sep 05 03:09:59.113071 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:09:59.112955    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" event={"ID":"e68d0f19-09ee-4980-90e9-49629ccc6fc6","Type":"ContainerStarted","Data":"91429d4fa8bbb7ea2fe9ce5c3d4da6f553150a10739e7ccb36426e58811ba346"}
Sep 05 03:10:02.129149 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:02.129116    2555 generic.go:356] "Generic (PLEG): container finished" podID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerID="dd7a449ac68569f2dd45ede2d616ece0a17aee68cf966caf57b4d3442be562da" exitCode=0
Sep 05 03:10:02.129606 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:02.129189    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" event={"ID":"e68d0f19-09ee-4980-90e9-49629ccc6fc6","Type":"ContainerDied","Data":"dd7a449ac68569f2dd45ede2d616ece0a17aee68cf966caf57b4d3442be562da"}
Sep 05 03:10:03.135592 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:03.135555    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" event={"ID":"e68d0f19-09ee-4980-90e9-49629ccc6fc6","Type":"ContainerStarted","Data":"b84e1a31564c986c011b97061c6ca73adbe01caef4f0cff285c33800b1198b9b"}
Sep 05 03:10:03.136035 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:03.135617    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" event={"ID":"e68d0f19-09ee-4980-90e9-49629ccc6fc6","Type":"ContainerStarted","Data":"ee12065709682b6c961fdefd67ec924861464648f5d78afc1f4ba025d3213e74"}
Sep 05 03:10:03.136035 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:03.135845    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:10:03.157461 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:03.157405    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podStartSLOduration=2.363874586 podStartE2EDuration="6.157383457s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.793508871s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:10:03.155998276 +0000 UTC m=+3124.164494438" watchObservedRunningTime="2026-09-05 03:10:03.157383457 +0000 UTC m=+3124.165879597"
Sep 05 03:10:04.141490 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:04.141451    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:10:04.146285 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:04.146237    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 05 03:10:05.144924 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:05.144882    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 05 03:10:10.149905 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:10.149868    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:10:10.150421 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:10.150394    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 05 03:10:13.174890 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.174851    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"]
Sep 05 03:10:13.178374 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.178353    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.180664 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.180643    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6920a4-predictor-serving-cert\""
Sep 05 03:10:13.180782 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.180738    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6920a4-kube-rbac-proxy-sar-config\""
Sep 05 03:10:13.180782 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.180744    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-6920a4\""
Sep 05 03:10:13.180900 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.180857    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-6920a4-dockercfg-zx2tm\""
Sep 05 03:10:13.188915 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.188894    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"]
Sep 05 03:10:13.330102 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.330067    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-cabundle-cert\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.330281 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.330112    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dh5x\" (UniqueName: \"kubernetes.io/projected/6f704236-b01c-40a5-9846-9d69477aba77-kube-api-access-2dh5x\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.330281 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.330135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.330281 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.330262    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f704236-b01c-40a5-9846-9d69477aba77-proxy-tls\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.330388 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.330311    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f704236-b01c-40a5-9846-9d69477aba77-kserve-provision-location\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.431417 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.431322    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.431417 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.431396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f704236-b01c-40a5-9846-9d69477aba77-proxy-tls\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.431685 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.431423    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f704236-b01c-40a5-9846-9d69477aba77-kserve-provision-location\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.431685 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.431447    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-cabundle-cert\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.431685 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.431492    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dh5x\" (UniqueName: \"kubernetes.io/projected/6f704236-b01c-40a5-9846-9d69477aba77-kube-api-access-2dh5x\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.431973 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.431943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f704236-b01c-40a5-9846-9d69477aba77-kserve-provision-location\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.432164 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.432139    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.432237 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.432161    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-cabundle-cert\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.433925 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.433907    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f704236-b01c-40a5-9846-9d69477aba77-proxy-tls\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.439378 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.439355    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dh5x\" (UniqueName: \"kubernetes.io/projected/6f704236-b01c-40a5-9846-9d69477aba77-kube-api-access-2dh5x\") pod \"isvc-secondary-6920a4-predictor-84b59f87c-hrrdx\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") " pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.489429 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.489392    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:13.768065 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:13.768041    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"]
Sep 05 03:10:13.770211 ip-10-0-135-239 kubenswrapper[2555]: W0905 03:10:13.770181    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f704236_b01c_40a5_9846_9d69477aba77.slice/crio-9e3f51164d38f645e57122ff84e930030ffaf952130d4142b39a68a9d21fc9ec WatchSource:0}: Error finding container 9e3f51164d38f645e57122ff84e930030ffaf952130d4142b39a68a9d21fc9ec: Status 404 returned error can't find the container with id 9e3f51164d38f645e57122ff84e930030ffaf952130d4142b39a68a9d21fc9ec
Sep 05 03:10:14.186550 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:14.186502    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx" event={"ID":"6f704236-b01c-40a5-9846-9d69477aba77","Type":"ContainerStarted","Data":"6da686679680bd73073aefd54d4224b25f5875e5ac8c22bedb5c6fd9f2d48300"}
Sep 05 03:10:14.186909 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:14.186631    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx" event={"ID":"6f704236-b01c-40a5-9846-9d69477aba77","Type":"ContainerStarted","Data":"9e3f51164d38f645e57122ff84e930030ffaf952130d4142b39a68a9d21fc9ec"}
Sep 05 03:10:18.204675 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:18.204644    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6920a4-predictor-84b59f87c-hrrdx_6f704236-b01c-40a5-9846-9d69477aba77/storage-initializer/0.log"
Sep 05 03:10:18.205041 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:18.204685    2555 generic.go:356] "Generic (PLEG): container finished" podID="6f704236-b01c-40a5-9846-9d69477aba77" containerID="6da686679680bd73073aefd54d4224b25f5875e5ac8c22bedb5c6fd9f2d48300" exitCode=1
Sep 05 03:10:18.205041 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:18.204766    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx" event={"ID":"6f704236-b01c-40a5-9846-9d69477aba77","Type":"ContainerDied","Data":"6da686679680bd73073aefd54d4224b25f5875e5ac8c22bedb5c6fd9f2d48300"}
Sep 05 03:10:19.210843 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:19.210812    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6920a4-predictor-84b59f87c-hrrdx_6f704236-b01c-40a5-9846-9d69477aba77/storage-initializer/0.log"
Sep 05 03:10:19.211248 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:19.210869    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx" event={"ID":"6f704236-b01c-40a5-9846-9d69477aba77","Type":"ContainerStarted","Data":"9d0b383723bec19dd09f9599f6616f6867a941aac66ba18eca94f38ffcc7a760"}
Sep 05 03:10:20.150333 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:20.150290    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 05 03:10:22.224921 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:22.224843    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6920a4-predictor-84b59f87c-hrrdx_6f704236-b01c-40a5-9846-9d69477aba77/storage-initializer/1.log"
Sep 05 03:10:22.225308 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:22.225197    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6920a4-predictor-84b59f87c-hrrdx_6f704236-b01c-40a5-9846-9d69477aba77/storage-initializer/0.log"
Sep 05 03:10:22.225308 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:22.225229    2555 generic.go:356] "Generic (PLEG): container finished" podID="6f704236-b01c-40a5-9846-9d69477aba77" containerID="9d0b383723bec19dd09f9599f6616f6867a941aac66ba18eca94f38ffcc7a760" exitCode=1
Sep 05 03:10:22.225308 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:22.225279    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx" event={"ID":"6f704236-b01c-40a5-9846-9d69477aba77","Type":"ContainerDied","Data":"9d0b383723bec19dd09f9599f6616f6867a941aac66ba18eca94f38ffcc7a760"}
Sep 05 03:10:22.225434 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:22.225338    2555 scope.go:117] "RemoveContainer" containerID="6da686679680bd73073aefd54d4224b25f5875e5ac8c22bedb5c6fd9f2d48300"
Sep 05 03:10:22.225714 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:22.225688    2555 scope.go:117] "RemoveContainer" containerID="6da686679680bd73073aefd54d4224b25f5875e5ac8c22bedb5c6fd9f2d48300"
Sep 05 03:10:22.236642 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:10:22.236616    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-6920a4-predictor-84b59f87c-hrrdx_kserve-ci-e2e-test(6f704236-b01c-40a5-9846-9d69477aba77)\"" pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx" podUID="6f704236-b01c-40a5-9846-9d69477aba77"
Sep 05 03:10:22.236847 ip-10-0-135-239 kubenswrapper[2555]: E0905 03:10:22.236817    2555 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6920a4-predictor-84b59f87c-hrrdx_kserve-ci-e2e-test_6f704236-b01c-40a5-9846-9d69477aba77_0 in pod sandbox 9e3f51164d38f645e57122ff84e930030ffaf952130d4142b39a68a9d21fc9ec from index: no such id: '6da686679680bd73073aefd54d4224b25f5875e5ac8c22bedb5c6fd9f2d48300'" containerID="6da686679680bd73073aefd54d4224b25f5875e5ac8c22bedb5c6fd9f2d48300"
Sep 05 03:10:22.236945 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:22.236851    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da686679680bd73073aefd54d4224b25f5875e5ac8c22bedb5c6fd9f2d48300"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6920a4-predictor-84b59f87c-hrrdx_kserve-ci-e2e-test_6f704236-b01c-40a5-9846-9d69477aba77_0 in pod sandbox 9e3f51164d38f645e57122ff84e930030ffaf952130d4142b39a68a9d21fc9ec from index: no such id: '6da686679680bd73073aefd54d4224b25f5875e5ac8c22bedb5c6fd9f2d48300'"
Sep 05 03:10:23.230612 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:23.230582    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6920a4-predictor-84b59f87c-hrrdx_6f704236-b01c-40a5-9846-9d69477aba77/storage-initializer/1.log"
Sep 05 03:10:29.213481 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.213449    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"]
Sep 05 03:10:29.351144 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.351115    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6920a4-predictor-84b59f87c-hrrdx_6f704236-b01c-40a5-9846-9d69477aba77/storage-initializer/1.log"
Sep 05 03:10:29.355026 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.355004    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:29.364774 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.364744    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f704236-b01c-40a5-9846-9d69477aba77-kserve-provision-location\") pod \"6f704236-b01c-40a5-9846-9d69477aba77\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") "
Sep 05 03:10:29.364944 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.364789    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-cabundle-cert\") pod \"6f704236-b01c-40a5-9846-9d69477aba77\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") "
Sep 05 03:10:29.364944 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.364815    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f704236-b01c-40a5-9846-9d69477aba77-proxy-tls\") pod \"6f704236-b01c-40a5-9846-9d69477aba77\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") "
Sep 05 03:10:29.364944 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.364885    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-kube-rbac-proxy-sar-config\") pod \"6f704236-b01c-40a5-9846-9d69477aba77\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") "
Sep 05 03:10:29.364944 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.364919    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dh5x\" (UniqueName: \"kubernetes.io/projected/6f704236-b01c-40a5-9846-9d69477aba77-kube-api-access-2dh5x\") pod \"6f704236-b01c-40a5-9846-9d69477aba77\" (UID: \"6f704236-b01c-40a5-9846-9d69477aba77\") "
Sep 05 03:10:29.365155 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.365035    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f704236-b01c-40a5-9846-9d69477aba77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f704236-b01c-40a5-9846-9d69477aba77" (UID: "6f704236-b01c-40a5-9846-9d69477aba77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:10:29.365222 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.365199    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f704236-b01c-40a5-9846-9d69477aba77" (UID: "6f704236-b01c-40a5-9846-9d69477aba77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:10:29.365292 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.365262    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f704236-b01c-40a5-9846-9d69477aba77-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:10:29.365342 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.365297    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f704236-b01c-40a5-9846-9d69477aba77" (UID: "6f704236-b01c-40a5-9846-9d69477aba77"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:10:29.367011 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.366992    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f704236-b01c-40a5-9846-9d69477aba77-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f704236-b01c-40a5-9846-9d69477aba77" (UID: "6f704236-b01c-40a5-9846-9d69477aba77"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:10:29.367094 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.367074    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f704236-b01c-40a5-9846-9d69477aba77-kube-api-access-2dh5x" (OuterVolumeSpecName: "kube-api-access-2dh5x") pod "6f704236-b01c-40a5-9846-9d69477aba77" (UID: "6f704236-b01c-40a5-9846-9d69477aba77"). InnerVolumeSpecName "kube-api-access-2dh5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:10:29.465763 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.465677    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:10:29.465763 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.465706    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f704236-b01c-40a5-9846-9d69477aba77-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:10:29.465763 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.465715    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f704236-b01c-40a5-9846-9d69477aba77-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:10:29.465763 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:29.465728    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dh5x\" (UniqueName: \"kubernetes.io/projected/6f704236-b01c-40a5-9846-9d69477aba77-kube-api-access-2dh5x\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:10:30.151235 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:30.151193    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 05 03:10:30.264988 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:30.264958    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6920a4-predictor-84b59f87c-hrrdx_6f704236-b01c-40a5-9846-9d69477aba77/storage-initializer/1.log"
Sep 05 03:10:30.265370 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:30.265058    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"
Sep 05 03:10:30.265370 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:30.265075    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx" event={"ID":"6f704236-b01c-40a5-9846-9d69477aba77","Type":"ContainerDied","Data":"9e3f51164d38f645e57122ff84e930030ffaf952130d4142b39a68a9d21fc9ec"}
Sep 05 03:10:30.265370 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:30.265139    2555 scope.go:117] "RemoveContainer" containerID="9d0b383723bec19dd09f9599f6616f6867a941aac66ba18eca94f38ffcc7a760"
Sep 05 03:10:30.311515 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:30.311485    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"]
Sep 05 03:10:30.319236 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:30.319208    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6920a4-predictor-84b59f87c-hrrdx"]
Sep 05 03:10:31.544049 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:31.544014    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f704236-b01c-40a5-9846-9d69477aba77" path="/var/lib/kubelet/pods/6f704236-b01c-40a5-9846-9d69477aba77/volumes"
Sep 05 03:10:40.150688 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:40.150646    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 05 03:10:50.150368 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:10:50.150324    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 05 03:11:00.150679 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:00.150639    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 05 03:11:10.150974 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:10.150927    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 05 03:11:15.538557 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:15.538487    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 05 03:11:25.543975 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:25.543945    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:11:27.974631 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:27.974601    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"]
Sep 05 03:11:27.975019 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:27.974826    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container" containerID="cri-o://ee12065709682b6c961fdefd67ec924861464648f5d78afc1f4ba025d3213e74" gracePeriod=30
Sep 05 03:11:27.975019 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:27.974909    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kube-rbac-proxy" containerID="cri-o://b84e1a31564c986c011b97061c6ca73adbe01caef4f0cff285c33800b1198b9b" gracePeriod=30
Sep 05 03:11:28.526248 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:28.526206    2555 generic.go:356] "Generic (PLEG): container finished" podID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerID="b84e1a31564c986c011b97061c6ca73adbe01caef4f0cff285c33800b1198b9b" exitCode=2
Sep 05 03:11:28.526461 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:28.526277    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" event={"ID":"e68d0f19-09ee-4980-90e9-49629ccc6fc6","Type":"ContainerDied","Data":"b84e1a31564c986c011b97061c6ca73adbe01caef4f0cff285c33800b1198b9b"}
Sep 05 03:11:30.145276 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:30.145230    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Sep 05 03:11:31.541763 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.541726    2555 generic.go:356] "Generic (PLEG): container finished" podID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerID="ee12065709682b6c961fdefd67ec924861464648f5d78afc1f4ba025d3213e74" exitCode=0
Sep 05 03:11:31.543816 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.543785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" event={"ID":"e68d0f19-09ee-4980-90e9-49629ccc6fc6","Type":"ContainerDied","Data":"ee12065709682b6c961fdefd67ec924861464648f5d78afc1f4ba025d3213e74"}
Sep 05 03:11:31.629596 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.629566    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:11:31.700018 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.699935    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-rbac-proxy-sar-config\") pod \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") "
Sep 05 03:11:31.700018 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.699973    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kserve-provision-location\") pod \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") "
Sep 05 03:11:31.700246 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.700027    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-cabundle-cert\") pod \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") "
Sep 05 03:11:31.700246 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.700069    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68d0f19-09ee-4980-90e9-49629ccc6fc6-proxy-tls\") pod \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") "
Sep 05 03:11:31.700246 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.700097    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcr8g\" (UniqueName: \"kubernetes.io/projected/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-api-access-bcr8g\") pod \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\" (UID: \"e68d0f19-09ee-4980-90e9-49629ccc6fc6\") "
Sep 05 03:11:31.700404 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.700339    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e68d0f19-09ee-4980-90e9-49629ccc6fc6" (UID: "e68d0f19-09ee-4980-90e9-49629ccc6fc6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:11:31.700404 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.700354    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e68d0f19-09ee-4980-90e9-49629ccc6fc6" (UID: "e68d0f19-09ee-4980-90e9-49629ccc6fc6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:11:31.700404 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.700365    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e68d0f19-09ee-4980-90e9-49629ccc6fc6" (UID: "e68d0f19-09ee-4980-90e9-49629ccc6fc6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:11:31.702170 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.702137    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-api-access-bcr8g" (OuterVolumeSpecName: "kube-api-access-bcr8g") pod "e68d0f19-09ee-4980-90e9-49629ccc6fc6" (UID: "e68d0f19-09ee-4980-90e9-49629ccc6fc6"). InnerVolumeSpecName "kube-api-access-bcr8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:11:31.702277 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.702172    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68d0f19-09ee-4980-90e9-49629ccc6fc6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e68d0f19-09ee-4980-90e9-49629ccc6fc6" (UID: "e68d0f19-09ee-4980-90e9-49629ccc6fc6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:11:31.801412 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.801363    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcr8g\" (UniqueName: \"kubernetes.io/projected/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-api-access-bcr8g\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:11:31.801412 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.801407    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:11:31.801412 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.801418    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e68d0f19-09ee-4980-90e9-49629ccc6fc6-kserve-provision-location\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:11:31.801412 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.801427    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e68d0f19-09ee-4980-90e9-49629ccc6fc6-cabundle-cert\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:11:31.801724 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:31.801437    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68d0f19-09ee-4980-90e9-49629ccc6fc6-proxy-tls\") on node \"ip-10-0-135-239.ec2.internal\" DevicePath \"\""
Sep 05 03:11:32.548254 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:32.548224    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"
Sep 05 03:11:32.548777 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:32.548254    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm" event={"ID":"e68d0f19-09ee-4980-90e9-49629ccc6fc6","Type":"ContainerDied","Data":"91429d4fa8bbb7ea2fe9ce5c3d4da6f553150a10739e7ccb36426e58811ba346"}
Sep 05 03:11:32.548777 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:32.548316    2555 scope.go:117] "RemoveContainer" containerID="b84e1a31564c986c011b97061c6ca73adbe01caef4f0cff285c33800b1198b9b"
Sep 05 03:11:32.557043 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:32.557023    2555 scope.go:117] "RemoveContainer" containerID="ee12065709682b6c961fdefd67ec924861464648f5d78afc1f4ba025d3213e74"
Sep 05 03:11:32.564229 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:32.564211    2555 scope.go:117] "RemoveContainer" containerID="dd7a449ac68569f2dd45ede2d616ece0a17aee68cf966caf57b4d3442be562da"
Sep 05 03:11:32.568952 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:32.568930    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"]
Sep 05 03:11:32.572663 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:32.572645    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hnwcm"]
Sep 05 03:11:33.543686 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:11:33.543655    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" path="/var/lib/kubelet/pods/e68d0f19-09ee-4980-90e9-49629ccc6fc6/volumes"
Sep 05 03:12:11.335732 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:11.335703    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8dkfv_bd509ebe-1474-4b42-88ae-96220ceecee8/global-pull-secret-syncer/0.log"
Sep 05 03:12:11.488725 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:11.488693    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-96mr6_89c89ec9-f23a-4b39-83ed-d04f5c12362e/konnectivity-agent/0.log"
Sep 05 03:12:11.576493 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:11.576458    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-239.ec2.internal_1ee890bde32bbb8ee31046089c37406d/haproxy/0.log"
Sep 05 03:12:15.539710 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.539684    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6/alertmanager/0.log"
Sep 05 03:12:15.568966 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.568941    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6/config-reloader/0.log"
Sep 05 03:12:15.595084 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.595057    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6/kube-rbac-proxy-web/0.log"
Sep 05 03:12:15.622715 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.622688    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6/kube-rbac-proxy/0.log"
Sep 05 03:12:15.646474 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.646445    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6/kube-rbac-proxy-metric/0.log"
Sep 05 03:12:15.668512 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.668485    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6/prom-label-proxy/0.log"
Sep 05 03:12:15.691981 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.691962    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bcb9e01c-a94b-4ce9-9690-5e16a6fd5ce6/init-config-reloader/0.log"
Sep 05 03:12:15.767161 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.767132    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-nt84q_3dc1bfa3-ce6a-4794-911b-959cadd385a4/kube-state-metrics/0.log"
Sep 05 03:12:15.790114 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.790037    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-nt84q_3dc1bfa3-ce6a-4794-911b-959cadd385a4/kube-rbac-proxy-main/0.log"
Sep 05 03:12:15.811584 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.811560    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-nt84q_3dc1bfa3-ce6a-4794-911b-959cadd385a4/kube-rbac-proxy-self/0.log"
Sep 05 03:12:15.896743 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.896716    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hxvj_4259cc42-4148-48c0-917d-b3e88155d4a2/node-exporter/0.log"
Sep 05 03:12:15.920911 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.920885    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hxvj_4259cc42-4148-48c0-917d-b3e88155d4a2/kube-rbac-proxy/0.log"
Sep 05 03:12:15.943444 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:15.943421    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hxvj_4259cc42-4148-48c0-917d-b3e88155d4a2/init-textfile/0.log"
Sep 05 03:12:16.420844 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:16.420815    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-cvshl_0cfb41ee-1e4c-4673-87ff-58216a84edd8/prometheus-operator/0.log"
Sep 05 03:12:16.439074 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:16.439054    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-cvshl_0cfb41ee-1e4c-4673-87ff-58216a84edd8/kube-rbac-proxy/0.log"
Sep 05 03:12:16.494467 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:16.494443    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-789c6dcd94-nqlnn_cc60118c-39a7-4b03-9a78-8f1556aa58ca/telemeter-client/0.log"
Sep 05 03:12:16.516789 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:16.516767    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-789c6dcd94-nqlnn_cc60118c-39a7-4b03-9a78-8f1556aa58ca/reload/0.log"
Sep 05 03:12:16.536899 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:16.536870    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-789c6dcd94-nqlnn_cc60118c-39a7-4b03-9a78-8f1556aa58ca/kube-rbac-proxy/0.log"
Sep 05 03:12:18.013455 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:18.013418    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-x6s4p_b0178d25-ade8-4d57-bacc-137fc466bbce/networking-console-plugin/0.log"
Sep 05 03:12:18.550298 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:18.550209    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/1.log"
Sep 05 03:12:18.554730 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:18.554711    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vwlqs_30f1f819-6e52-4eb7-b546-47f22a4d5f8b/console-operator/2.log"
Sep 05 03:12:18.980665 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:18.980636    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-blnjb_c7611f52-960d-4175-a893-8c3efe168f2f/download-server/0.log"
Sep 05 03:12:19.336521 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336439    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"]
Sep 05 03:12:19.336893 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336780    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f704236-b01c-40a5-9846-9d69477aba77" containerName="storage-initializer"
Sep 05 03:12:19.336893 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336792    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f704236-b01c-40a5-9846-9d69477aba77" containerName="storage-initializer"
Sep 05 03:12:19.336893 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336800    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kube-rbac-proxy"
Sep 05 03:12:19.336893 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336806    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kube-rbac-proxy"
Sep 05 03:12:19.336893 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336818    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container"
Sep 05 03:12:19.336893 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336824    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container"
Sep 05 03:12:19.336893 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336831    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f704236-b01c-40a5-9846-9d69477aba77" containerName="storage-initializer"
Sep 05 03:12:19.336893 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336836    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f704236-b01c-40a5-9846-9d69477aba77" containerName="storage-initializer"
Sep 05 03:12:19.336893 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336845    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="storage-initializer"
Sep 05 03:12:19.336893 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336850    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="storage-initializer"
Sep 05 03:12:19.337231 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336907    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f704236-b01c-40a5-9846-9d69477aba77" containerName="storage-initializer"
Sep 05 03:12:19.337231 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336917    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kserve-container"
Sep 05 03:12:19.337231 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336922    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f704236-b01c-40a5-9846-9d69477aba77" containerName="storage-initializer"
Sep 05 03:12:19.337231 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.336930    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68d0f19-09ee-4980-90e9-49629ccc6fc6" containerName="kube-rbac-proxy"
Sep 05 03:12:19.339892 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.339876    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.342145 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.342121    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b5gnx\"/\"openshift-service-ca.crt\""
Sep 05 03:12:19.343085 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.343067    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b5gnx\"/\"kube-root-ca.crt\""
Sep 05 03:12:19.343167 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.343068    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b5gnx\"/\"default-dockercfg-ngc74\""
Sep 05 03:12:19.348502 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.348480    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"]
Sep 05 03:12:19.398835 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.398801    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-proc\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.399003 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.398843    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-lib-modules\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.399003 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.398873    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhf9v\" (UniqueName: \"kubernetes.io/projected/66f959ef-4f61-4019-967b-9bb2ad330c66-kube-api-access-dhf9v\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.399003 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.398934    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-podres\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.399003 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.398990    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-sys\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.499944 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.499906    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-proc\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.499944 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.499949    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-lib-modules\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.500147 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.499971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhf9v\" (UniqueName: \"kubernetes.io/projected/66f959ef-4f61-4019-967b-9bb2ad330c66-kube-api-access-dhf9v\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.500147 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.499991    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-podres\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.500147 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.500051    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-sys\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.500147 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.500048    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-proc\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.500147 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.500107    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-lib-modules\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.500147 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.500122    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-sys\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.500346 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.500189    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66f959ef-4f61-4019-967b-9bb2ad330c66-podres\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.508288 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.508260    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhf9v\" (UniqueName: \"kubernetes.io/projected/66f959ef-4f61-4019-967b-9bb2ad330c66-kube-api-access-dhf9v\") pod \"perf-node-gather-daemonset-bz584\" (UID: \"66f959ef-4f61-4019-967b-9bb2ad330c66\") " pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.513687 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.513666    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-l9d7p_ba09a841-916f-435d-8f70-221248e93a65/volume-data-source-validator/0.log"
Sep 05 03:12:19.650367 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.650336    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:19.952174 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:19.952148    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"]
Sep 05 03:12:19.954430 ip-10-0-135-239 kubenswrapper[2555]: W0905 03:12:19.954405    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod66f959ef_4f61_4019_967b_9bb2ad330c66.slice/crio-293e240b8e70bfc67af24f188917274715ce9586a25b76ccbb5706484271cc4d WatchSource:0}: Error finding container 293e240b8e70bfc67af24f188917274715ce9586a25b76ccbb5706484271cc4d: Status 404 returned error can't find the container with id 293e240b8e70bfc67af24f188917274715ce9586a25b76ccbb5706484271cc4d
Sep 05 03:12:20.392760 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:20.392731    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9l9d9_f0769ef0-90d9-43ee-ade4-5aae6141a935/dns/0.log"
Sep 05 03:12:20.417386 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:20.417358    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9l9d9_f0769ef0-90d9-43ee-ade4-5aae6141a935/kube-rbac-proxy/0.log"
Sep 05 03:12:20.582518 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:20.582494    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rfzkh_09e1b36b-9b93-4340-a648-fafc7e0b6f90/dns-node-resolver/0.log"
Sep 05 03:12:20.764921 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:20.764824    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584" event={"ID":"66f959ef-4f61-4019-967b-9bb2ad330c66","Type":"ContainerStarted","Data":"c42c535540ba84f47a5ffc828f891dddbe5443930cece55d28de9b3903785c39"}
Sep 05 03:12:20.764921 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:20.764881    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584" event={"ID":"66f959ef-4f61-4019-967b-9bb2ad330c66","Type":"ContainerStarted","Data":"293e240b8e70bfc67af24f188917274715ce9586a25b76ccbb5706484271cc4d"}
Sep 05 03:12:20.765111 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:20.764960    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:20.780983 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:20.780929    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584" podStartSLOduration=1.780912601 podStartE2EDuration="1.780912601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 03:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:12:20.779481076 +0000 UTC m=+3261.787977235" watchObservedRunningTime="2026-09-05 03:12:20.780912601 +0000 UTC m=+3261.789408740"
Sep 05 03:12:21.036632 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:21.036530    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7b55c9ff6-vs5m6_59c7dd2d-013b-430c-bd53-e60d895f2365/registry/0.log"
Sep 05 03:12:21.105661 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:21.105629    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8zqc9_06c658b0-5ad2-43c5-8057-633c5525d4ab/node-ca/0.log"
Sep 05 03:12:22.184590 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:22.184557    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rsph8_4f3f8d4a-ee71-418d-958e-3a92b0d250be/serve-healthcheck-canary/0.log"
Sep 05 03:12:22.636919 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:22.636893    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4cllv_6b2c3ca8-5f11-4fc2-a88c-70e8a498d693/kube-rbac-proxy/0.log"
Sep 05 03:12:22.656661 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:22.656633    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4cllv_6b2c3ca8-5f11-4fc2-a88c-70e8a498d693/exporter/0.log"
Sep 05 03:12:22.676825 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:22.676799    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4cllv_6b2c3ca8-5f11-4fc2-a88c-70e8a498d693/extractor/0.log"
Sep 05 03:12:25.223413 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:25.223386    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-sc9n4_3809d232-09b8-457f-81a9-d51b5c4d298e/manager/0.log"
Sep 05 03:12:25.889084 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:25.889057    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-ndq76_85381b15-07d1-433d-bf7b-7e6494eef6d8/seaweedfs-tls-serving/0.log"
Sep 05 03:12:26.779099 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:26.779071    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b5gnx/perf-node-gather-daemonset-bz584"
Sep 05 03:12:31.722878 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:31.722853    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb2xw_5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae/kube-multus-additional-cni-plugins/0.log"
Sep 05 03:12:31.747683 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:31.747651    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb2xw_5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae/egress-router-binary-copy/0.log"
Sep 05 03:12:31.773676 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:31.773652    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb2xw_5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae/cni-plugins/0.log"
Sep 05 03:12:31.793735 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:31.793706    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb2xw_5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae/bond-cni-plugin/0.log"
Sep 05 03:12:31.814279 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:31.814257    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb2xw_5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae/routeoverride-cni/0.log"
Sep 05 03:12:31.835685 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:31.835660    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb2xw_5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae/whereabouts-cni-bincopy/0.log"
Sep 05 03:12:31.865784 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:31.865763    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb2xw_5b5709cf-8c39-4bd7-9f0a-0ce5afed64ae/whereabouts-cni/0.log"
Sep 05 03:12:31.980900 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:31.980831    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xbqkm_57d95fd2-fff8-4df2-9ae3-89d120559276/kube-multus/0.log"
Sep 05 03:12:32.001487 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:32.001455    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-85xds_2dd23bdc-d907-4b00-95c5-24828e92da98/network-metrics-daemon/0.log"
Sep 05 03:12:32.021109 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:32.021085    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-85xds_2dd23bdc-d907-4b00-95c5-24828e92da98/kube-rbac-proxy/0.log"
Sep 05 03:12:32.863010 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:32.862978    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sg7d_d6059461-d098-4d3f-8ae4-11d5a1b09ff5/ovn-controller/0.log"
Sep 05 03:12:32.879328 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:32.879308    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sg7d_d6059461-d098-4d3f-8ae4-11d5a1b09ff5/ovn-acl-logging/0.log"
Sep 05 03:12:32.900075 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:32.900055    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sg7d_d6059461-d098-4d3f-8ae4-11d5a1b09ff5/kube-rbac-proxy-node/0.log"
Sep 05 03:12:32.920613 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:32.920592    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sg7d_d6059461-d098-4d3f-8ae4-11d5a1b09ff5/kube-rbac-proxy-ovn-metrics/0.log"
Sep 05 03:12:32.940181 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:32.940149    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sg7d_d6059461-d098-4d3f-8ae4-11d5a1b09ff5/northd/0.log"
Sep 05 03:12:32.960709 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:32.960685    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sg7d_d6059461-d098-4d3f-8ae4-11d5a1b09ff5/nbdb/0.log"
Sep 05 03:12:32.979879 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:32.979859    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sg7d_d6059461-d098-4d3f-8ae4-11d5a1b09ff5/sbdb/0.log"
Sep 05 03:12:33.020509 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:33.020485    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2sg7d_d6059461-d098-4d3f-8ae4-11d5a1b09ff5/ovnkube-controller/0.log"
Sep 05 03:12:34.338890 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:34.338865    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-cbsx2_0664d7c3-d7ef-41b5-97f5-8e9fd173da99/check-endpoints/0.log"
Sep 05 03:12:34.363156 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:34.363134    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8t685_df75f0b8-d51e-4c43-a501-58dc006fef8e/network-check-target-container/0.log"
Sep 05 03:12:35.318923 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:35.318894    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-bx78n_7b8c1413-db22-4560-9991-6260629f38be/iptables-alerter/0.log"
Sep 05 03:12:36.085504 ip-10-0-135-239 kubenswrapper[2555]: I0905 03:12:36.085474    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vtds2_897e2168-985a-4ac7-95eb-9eb7cf09aa12/tuned/0.log"