Sep 17 16:41:29.994555 ip-10-0-131-202 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 16:41:30.416041 ip-10-0-131-202 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:30.416041 ip-10-0-131-202 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 16:41:30.416041 ip-10-0-131-202 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:30.416041 ip-10-0-131-202 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 16:41:30.416041 ip-10-0-131-202 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:30.417610 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.417532    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 16:41:30.423303 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423288    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:30.423303 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423303    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423319    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423323    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423328    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423333    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423336    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423339    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423342    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423345    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423348    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423351    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423354    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423357    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423364    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423367    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423370    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423372    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423374    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423377    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:30.423379 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423380    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423383    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423385    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423388    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423391    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423394    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423397    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423399    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423402    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423405    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423408    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423410    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423413    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423415    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423417    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423420    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423422    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423425    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423428    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:30.423828 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423431    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423435    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423438    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423442    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423445    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423448    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423450    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423453    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423456    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423458    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423461    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423463    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423466    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423468    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423471    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423474    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423477    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423479    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423483    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:30.424285 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423486    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423489    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423492    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423495    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423497    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423500    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423503    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423505    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423508    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423511    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423513    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423516    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423518    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423525    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423527    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423530    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423533    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423535    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423538    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423540    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:30.424771 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423543    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423545    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423548    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423550    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423553    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423555    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423558    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423560    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423566    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423568    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423571    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423573    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423576    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423578    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423581    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423583    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423586    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423588    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423591    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423594    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:30.425256 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.423596    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424291    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424297    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424301    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424304    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424322    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424331    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424334    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424337    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424339    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424342    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424345    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424348    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424351    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424353    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424357    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424359    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424362    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424365    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:30.425785 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424367    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424371    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424374    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424377    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424380    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424382    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424385    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424388    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424391    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424393    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424396    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424399    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424401    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424404    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424406    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424409    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424411    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424414    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424417    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424420    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:30.426252 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424423    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424426    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424428    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424431    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424433    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424436    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424438    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424441    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424443    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424446    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424449    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424454    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424458    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424461    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424464    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424467    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424469    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424472    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424475    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:30.426757 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424477    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424481    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424483    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424486    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424488    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424491    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424493    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424496    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424500    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424504    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424507    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424511    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424514    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424517    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424520    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424523    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424526    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424529    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424531    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:30.427229 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424534    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424538    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424541    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424544    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424546    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424549    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424552    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424555    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424558    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424561    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424563    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424566    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424569    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424571    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424574    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424577    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424579    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424582    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424584    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:30.427716 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424587    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424590    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424593    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424595    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424954    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424959    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424962    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424965    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424968    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424971    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424973    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424976    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424979    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424981    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424985    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424988    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424991    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424993    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424996    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:30.428170 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.424999    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425002    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425005    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425008    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425010    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425013    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425016    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425018    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425021    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425023    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425026    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425028    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425031    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425033    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425036    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425039    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425041    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425044    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425048    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:30.428645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425051    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425053    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425056    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425059    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425062    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425065    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425067    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425070    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425072    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425075    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425078    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425080    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425083    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425087    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425092    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425095    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425097    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425100    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425103    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425105    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:30.429131 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425108    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425110    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425113    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425115    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425118    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425120    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425123    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425125    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425128    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425130    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425133    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425137    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425140    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425142    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425145    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425147    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425150    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425153    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425156    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425159    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:30.429634 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425162    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425164    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425166    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425170    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425173    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425176    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425181    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425184    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425187    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425190    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425192    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425195    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425198    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425201    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425203    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425206    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425208    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425211    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425214    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:30.430123 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425216    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425218    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425221    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425224    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425228    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425230    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425291    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425335    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425341    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425345    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425350    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425354    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425358    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425362    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425365    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425368    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425371    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425375    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425378    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425383    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425386    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425390    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425393    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425395    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 16:41:30.430645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425400    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425403    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425406    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425409    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425412    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425416    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425419    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425422    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425425    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425429    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425431    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425434    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425439    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425442    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425446    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425449    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425452    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425454    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425457    2566 flags.go:64] FLAG: --enable-server="true"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425460    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425464    2566 flags.go:64] FLAG: --event-burst="100"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425467    2566 flags.go:64] FLAG: --event-qps="50"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425470    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425473    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 16:41:30.431207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425476    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425480    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425482    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425487    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425490    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425493    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425496    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425499    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425501    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425504    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425507    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425510    2566 flags.go:64] FLAG: --feature-gates=""
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425513    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425516    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425519    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425522    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425525    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425528    2566 flags.go:64] FLAG: --help="false"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425530    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425534    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425538    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425541    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425545    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 16:41:30.431792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425548    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425551    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425554    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425556    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425559    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425562    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425565    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425568    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425571    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425573    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425577    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425579    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425583    2566 flags.go:64] FLAG: --lock-file=""
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425586    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425589    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425592    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425597    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425599    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425602    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425605    2566 flags.go:64] FLAG: --logging-format="text"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425608    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425611    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425614    2566 flags.go:64] FLAG: --manifest-url=""
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425616    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425621    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 16:41:30.432401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425624    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425627    2566 flags.go:64] FLAG: --max-pods="110"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425630    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425633    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425637    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425640    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425643    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425645    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425648    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425655    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425657    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425660    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425663    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425667    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425671    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425674    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425677    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425680    2566 flags.go:64] FLAG: --port="10250"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425683    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425685    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05caea108af26c45b"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425690    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425692    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425695    2566 flags.go:64] FLAG: --register-node="true"
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425698    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 17 16:41:30.433010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425702    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425704    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425707    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425710    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425713    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425716    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425719    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425722    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425725    2566 flags.go:64] FLAG: --runonce="false"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425727    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425730    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425733    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425737    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425740    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425743    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425746    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425749    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425752    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425755    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425757    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425760    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425763    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425766    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425769    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425774    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 16:41:30.433625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425777    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425779    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425783    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425787    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425790    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425793    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425796    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425799    2566 flags.go:64] FLAG: --v="2"
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425803    2566 flags.go:64] FLAG: --version="false"
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425806    2566 flags.go:64] FLAG: --vmodule=""
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425810    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.425813    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425916    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425919    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425923    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425926    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425929    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425932    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425935    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425938    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425941    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425944    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425947    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:30.434366 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425950    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425953    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425955    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425958    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425961    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425963    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425966    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425968    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425971    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425974    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425976    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425979    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425983    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425986    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425988    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425991    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425993    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425996    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.425999    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426001    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:30.434947 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426004    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426006    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426009    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426012    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426014    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426017    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426020    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426022    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426025    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426028    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426030    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426033    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426036    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426040    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426044    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426047    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426049    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426052    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426055    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:30.435447 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426058    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426062    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426065    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426067    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426070    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426074    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426076    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426079    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426082    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426084    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426087    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426090    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426092    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426095    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426097    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426100    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426103    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426106    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426108    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426111    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:30.435938 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426113    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426116    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426119    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426121    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426124    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426127    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426129    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426132    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426134    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426137    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426139    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426142    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426145    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426147    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426150    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426153    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426155    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426159    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426162    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426164    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:30.436438 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426167    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426170    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426172    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426175    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426178    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426182    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426185    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426188    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.426191    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.426928    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.434099    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.434111    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.435856    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.435864    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 16:41:30.436928 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435918    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435923    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435927    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435930    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435934    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435936    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435940    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435942    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435945    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435948    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435951    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435954    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435957    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435960    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435962    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435965    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435968    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435971    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435973    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435976    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:30.437280 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435980    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435983    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435986    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435989    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435991    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435994    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435997    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.435999    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436002    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436006    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436009    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436012    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436014    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436017    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436019    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436022    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436025    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436028    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436030    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436033    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:30.437817 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436035    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436038    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436041    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436043    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436046    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436048    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436051    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436055    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436058    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436061    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436064    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436067    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436069    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436072    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436074    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436077    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436080    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436082    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:30.438306 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436086    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436090    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436094    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436097    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436100    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436103    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436106    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436109    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436112    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436114    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436117    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436120    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436122    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436125    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436128    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436131    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436134    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436136    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436139    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436141    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:30.438759 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436145    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436147    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436150    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436153    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436155    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436158    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436161    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436163    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436166    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436169    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436172    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436174    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436177    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436179    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436182    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436185    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436187    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436190    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436192    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436195    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:30.439240 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436197    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.436202    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436303    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436320    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436324    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436327    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436330    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436333    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436336    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436338    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436341    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436344    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436347    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436351    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436353    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:30.439726 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436356    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436359    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436361    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436364    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436366    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436369    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436372    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436374    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436377    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436380    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436382    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436385    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436388    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436390    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436393    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436395    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436398    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436401    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436403    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436406    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:30.440097 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436409    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436411    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436414    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436416    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436419    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436421    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436424    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436426    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436429    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436431    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436434    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436438    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436442    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436445    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436448    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436451    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436454    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436456    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436459    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:30.440622 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436461    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436464    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436467    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436469    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436472    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436475    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436477    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436480    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436482    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436485    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436487    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436490    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436492    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436495    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436497    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436500    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436502    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436506    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436510    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436512    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:30.441084 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436515    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436518    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436522    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436525    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436528    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436531    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436533    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436536    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436538    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436541    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436544    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436546    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436548    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436551    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436554    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436557    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436559    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436562    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436565    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:30.441578 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436568    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436570    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436573    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436575    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436578    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436580    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436582    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:30.436585    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.436590    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.436690    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.438893    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.439892    2566 server.go:1019] "Starting client certificate rotation"
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.439995    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:41:30.442023 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.440057    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:41:30.442708 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.442694    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:41:30.446951 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.446930    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:41:30.457416 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.457400    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 17 16:41:30.459163 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.459149    2566 log.go:25] "Validated CRI v1 image API"
Sep 17 16:41:30.460041 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.460023    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 16:41:30.467950 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.467931    2566 fs.go:135] Filesystem UUIDs: map[60cc4d2a-04b7-4315-9dab-115ce6a516a7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bf4dedb4-94de-45e3-8d7c-59a0b43fd0bd:/dev/nvme0n1p3]
Sep 17 16:41:30.468013 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.467948    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 16:41:30.470919 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.470901    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:41:30.472822 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.472721    2566 manager.go:216] Machine: {Timestamp:2026-09-17 16:41:30.471364437 +0000 UTC m=+0.369312623 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101110 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c6eb0d0aa2300b6a8535d33530cdd SystemUUID:ec2c6eb0-d0aa-2300-b6a8-535d33530cdd BootID:75659b2d-b8ce-45e5-9b06-2c758c20903c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:e9:49:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:e9:49:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:7e:e9:2d:29:23 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 16:41:30.472822 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.472814    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 16:41:30.472945 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.472880    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 16:41:30.476644 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.476623    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 16:41:30.476765 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.476647    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-202.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 16:41:30.478173 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.478163    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 16:41:30.478205 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.478176    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 16:41:30.478205 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.478184    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:41:30.478205 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.478198    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:41:30.478293 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.478260    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 16:41:30.483410 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.483400    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:41:30.483542 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.483532    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 16:41:30.483572 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.483561    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 16:41:30.483602 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.483575    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 16:41:30.483602 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.483589    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 16:41:30.483602 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.483598    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 17 16:41:30.483705 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.483612    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 16:41:30.483905 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.483894    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:41:30.483942 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.483913    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:41:30.487983 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.487955    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 16:41:30.493147 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.493126    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 16:41:30.494761 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.494747    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnk4z"
Sep 17 16:41:30.499675 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.499661    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 16:41:30.499736 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.499690    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 16:41:30.500161 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.500148    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnk4z"
Sep 17 16:41:30.501353 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.501333    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 16:41:30.501858 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.501843    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 16:41:30.501934 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.501864    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 16:41:30.501934 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.501870    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 16:41:30.501934 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.501875    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 16:41:30.501934 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.501885    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 16:41:30.501934 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.501893    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 16:41:30.501934 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.501911    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 16:41:30.501934 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.501917    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 16:41:30.502225 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.502049    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 16:41:30.502225 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.502074    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 16:41:30.502225 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.502166    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 16:41:30.502225 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.502184    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 16:41:30.503877 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.503860    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 16:41:30.503877 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.503878    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 16:41:30.507773 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.507759    2566 server.go:1268] "Started kubelet"
Sep 17 16:41:30.507903 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.507864    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 16:41:30.507979 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.507891    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 16:41:30.507979 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.507925    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 16:41:30.508448 ip-10-0-131-202 systemd[1]: Started Kubernetes Kubelet.
Sep 17 16:41:30.510180 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.510163    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 16:41:30.510266 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.510242    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 16:41:30.517245 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.517219    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 16:41:30.519898 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.519884    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 16:41:30.519981 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.519899    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 16:41:30.520029 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.519961    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:30.520648 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.520632    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 16:41:30.520648 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.520633    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 16:41:30.520783 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.520668    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 16:41:30.520783 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.520729    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 16:41:30.520783 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.520740    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 16:41:30.520783 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.520754    2566 factory.go:55] Registering systemd factory
Sep 17 16:41:30.520783 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.520768    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 17 16:41:30.521011 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.520970    2566 factory.go:153] Registering CRI-O factory
Sep 17 16:41:30.521011 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.520986    2566 factory.go:223] Registration of the crio container factory successfully
Sep 17 16:41:30.521103 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.521035    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 16:41:30.521103 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.521055    2566 factory.go:103] Registering Raw factory
Sep 17 16:41:30.521103 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.521069    2566 manager.go:1196] Started watching for new ooms in manager
Sep 17 16:41:30.521441 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.521425    2566 manager.go:349] Starting recovery of all containers
Sep 17 16:41:30.523367 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.523345    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:30.523680 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.523659    2566 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-131-202.ec2.internal" not found
Sep 17 16:41:30.526046 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.525851    2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-202.ec2.internal\" not found" node="ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.534105 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.534089    2566 manager.go:354] Recovery completed
Sep 17 16:41:30.539464 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.539448    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:30.542057 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.542043    2566 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-131-202.ec2.internal" not found
Sep 17 16:41:30.543537 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.543519    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:30.543607 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.543549    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:30.543607 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.543561    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:30.544048 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.544034    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 16:41:30.544129 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.544054    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:30.544129 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.544054    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 16:41:30.544129 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.544092    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:41:30.544925 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.544906    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:30.544992 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.544936    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:30.544992 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.544950    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:30.546141 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.546128    2566 policy_none.go:49] "None policy: Start"
Sep 17 16:41:30.546210 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.546146    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 16:41:30.546210 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.546159    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 16:41:30.548082 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.548069    2566 policy_none.go:47] "Start"
Sep 17 16:41:30.598704 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.598689    2566 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-131-202.ec2.internal" not found
Sep 17 16:41:30.616393 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.616366    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 16:41:30.617490 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.617467    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 16:41:30.617563 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.617528    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 16:41:30.617563 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.617554    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 16:41:30.617652 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.617603    2566 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 17 16:41:30.619769 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.619750    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:30.620147 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.620134    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:30.715362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.715342    2566 manager.go:337] "Starting Device Plugin manager"
Sep 17 16:41:30.715452 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.715414    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 16:41:30.715452 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.715427    2566 server.go:87] "Starting device plugin registration server"
Sep 17 16:41:30.715676 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.715665    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 16:41:30.715712 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.715681    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 16:41:30.715790 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.715773    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 16:41:30.715864 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.715855    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 16:41:30.715864 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.715864    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 16:41:30.716348 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.716329    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 16:41:30.716420 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.716381    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:30.718490 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.718467    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-202.ec2.internal"]
Sep 17 16:41:30.718554 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.718542    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:30.719871 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.719857    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:30.719955 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.719887    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:30.719955 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.719902    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:30.722053 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.722038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ece54a7e142bdf40d1f2f69c78afd8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal\" (UID: \"3ece54a7e142bdf40d1f2f69c78afd8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.722150 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.722059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ece54a7e142bdf40d1f2f69c78afd8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal\" (UID: \"3ece54a7e142bdf40d1f2f69c78afd8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.722264 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.722251    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:30.722390 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.722374    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.722444 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.722404    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:30.722941 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.722927    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:30.723015 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.722959    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:30.723015 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.722970    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:30.723015 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.722982    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:30.723015 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.723013    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:30.723131 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.723026    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:30.725177 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.725162    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.725250 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.725185    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:30.725825 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.725807    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:30.725890 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.725835    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:30.725890 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.725845    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:30.758815 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.758796    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-202.ec2.internal\" not found" node="ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.762907 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.762893    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-202.ec2.internal\" not found" node="ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.816376 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.816238    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:30.817075 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.817055    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:30.817154 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.817083    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:30.817154 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.817093    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:30.817154 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.817111    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.822467 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.822453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ece54a7e142bdf40d1f2f69c78afd8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal\" (UID: \"3ece54a7e142bdf40d1f2f69c78afd8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.822528 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.822476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ece54a7e142bdf40d1f2f69c78afd8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal\" (UID: \"3ece54a7e142bdf40d1f2f69c78afd8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.822528 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.822495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c990e3bd2b767d7e89440c7596b76213-config\") pod \"kube-apiserver-proxy-ip-10-0-131-202.ec2.internal\" (UID: \"c990e3bd2b767d7e89440c7596b76213\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.822528 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.822524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ece54a7e142bdf40d1f2f69c78afd8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal\" (UID: \"3ece54a7e142bdf40d1f2f69c78afd8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.822629 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.822540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ece54a7e142bdf40d1f2f69c78afd8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal\" (UID: \"3ece54a7e142bdf40d1f2f69c78afd8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.826356 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.826335    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.826432 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.826358    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-202.ec2.internal\": node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:30.842638 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.842618    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:30.923471 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.923449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c990e3bd2b767d7e89440c7596b76213-config\") pod \"kube-apiserver-proxy-ip-10-0-131-202.ec2.internal\" (UID: \"c990e3bd2b767d7e89440c7596b76213\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.923551 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:30.923487    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c990e3bd2b767d7e89440c7596b76213-config\") pod \"kube-apiserver-proxy-ip-10-0-131-202.ec2.internal\" (UID: \"c990e3bd2b767d7e89440c7596b76213\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:30.943589 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:30.943549    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:31.044143 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:31.044115    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:31.062281 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:31.062257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:31.066603 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:31.066582    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:31.145106 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:31.145073    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:31.245687 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:31.245614    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:32.490704 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.490682    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:32.531988 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:32.531970    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:32.625517 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.625496    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 16:41:32.625892 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.625606    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 16:41:32.625892 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.625646    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 16:41:32.625892 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.625648    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 16:41:32.632781 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:32.632762    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:32.692185 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.692127    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 16:36:31 +0000 UTC" deadline="2028-04-24 13:18:56.774260473 +0000 UTC"
Sep 17 16:41:32.692185 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.692153    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14036h37m24.082109899s"
Sep 17 16:41:32.733681 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:32.733654    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-202.ec2.internal\" not found"
Sep 17 16:41:32.751584 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.751562    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:32.806322 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.806295    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:32.817352 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.817337    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:41:32.818336 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.818324    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-202.ec2.internal"
Sep 17 16:41:32.827428 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.827413    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:41:32.864543 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:32.864522    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc990e3bd2b767d7e89440c7596b76213.slice/crio-01057746c71ad58d0a811bfa4e08e497ef7faaac260be71f64e90b2796eb0371 WatchSource:0}: Error finding container 01057746c71ad58d0a811bfa4e08e497ef7faaac260be71f64e90b2796eb0371: Status 404 returned error can't find the container with id 01057746c71ad58d0a811bfa4e08e497ef7faaac260be71f64e90b2796eb0371
Sep 17 16:41:32.864826 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:32.864809    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ece54a7e142bdf40d1f2f69c78afd8b.slice/crio-a3a53eae1ea8338dfb02b8c291f0fccea9b3c61e1668f6d88ebc5061a08ff98e WatchSource:0}: Error finding container a3a53eae1ea8338dfb02b8c291f0fccea9b3c61e1668f6d88ebc5061a08ff98e: Status 404 returned error can't find the container with id a3a53eae1ea8338dfb02b8c291f0fccea9b3c61e1668f6d88ebc5061a08ff98e
Sep 17 16:41:32.869510 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:32.869497    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:41:33.463307 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.463280    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:33.670292 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.670267    2566 apiserver.go:52] "Watching apiserver"
Sep 17 16:41:33.678313 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.678203    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 16:41:33.679302 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.679277    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-qw97h","kube-system/konnectivity-agent-264lh","kube-system/kube-apiserver-proxy-ip-10-0-131-202.ec2.internal","openshift-dns/node-resolver-8hvh7","openshift-image-registry/node-ca-h774k","openshift-multus/multus-additional-cni-plugins-lkvgc","openshift-multus/network-metrics-daemon-vm27c","openshift-network-diagnostics/network-check-target-w6lnl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx","openshift-cluster-node-tuning-operator/tuned-82b45","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal","openshift-multus/multus-x4282","openshift-network-operator/iptables-alerter-r6nbk","openshift-ovn-kubernetes/ovnkube-node-cxtcx"]
Sep 17 16:41:33.681663 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.681635    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:33.681764 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:33.681716    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:33.683714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.683695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:33.686203 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.686032    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f5dkg\""
Sep 17 16:41:33.686203 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.686055    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 16:41:33.686203 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.686069    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 16:41:33.686884 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.686744    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8hvh7"
Sep 17 16:41:33.688966 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.688935    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 16:41:33.688966 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.688959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h774k"
Sep 17 16:41:33.689169 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.689153    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hb7x7\""
Sep 17 16:41:33.689366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.689346    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 16:41:33.691066 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.691045    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 16:41:33.691167 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.691130    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zt6c4\""
Sep 17 16:41:33.691225 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.691199    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.692274 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.691324    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 16:41:33.692274 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.691389    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 16:41:33.693811 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.693358    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 16:41:33.693811 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.693475    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 16:41:33.693811 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.693685    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 16:41:33.693811 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.693788    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 16:41:33.694066 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.693817    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:33.694066 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:33.693872    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:33.694177 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.694099    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 16:41:33.694405 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.694387    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xgdqh\""
Sep 17 16:41:33.695771 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.695751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:33.695873 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:33.695813    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:33.698084 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.698067    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.700566 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.700543    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 16:41:33.700657 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.700599    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 16:41:33.700657 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.700626    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4b7zt\""
Sep 17 16:41:33.700762 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.700729    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 16:41:33.700971 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.700954    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.702828 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.702804    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.703081 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.703059    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hkw8p\""
Sep 17 16:41:33.703352 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.703301    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:41:33.703451 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.703417    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 16:41:33.704900 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.704876    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 16:41:33.705326 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.705044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6nbk"
Sep 17 16:41:33.705326 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.705112    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2kp5h\""
Sep 17 16:41:33.707970 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.706480    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 16:41:33.707970 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.707714    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 16:41:33.707970 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.707807    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:41:33.707970 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.707941    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7246b\""
Sep 17 16:41:33.708190 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.708012    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 16:41:33.710647 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.710624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.714483 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.714416    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 16:41:33.714765 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.714445    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 16:41:33.715336 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.715114    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 16:41:33.715336 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.715130    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 16:41:33.715336 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.715176    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 16:41:33.715559 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.715379    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 16:41:33.715559 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.715439    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p9m68\""
Sep 17 16:41:33.717345 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.717327    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:41:33.717981 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.717959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5xmc\" (UniqueName: \"kubernetes.io/projected/dce7e7cb-88b0-4419-b2c9-144f94438855-kube-api-access-k5xmc\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.718075 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10-host\") pod \"node-ca-h774k\" (UID: \"89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10\") " pod="openshift-image-registry/node-ca-h774k"
Sep 17 16:41:33.718243 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718094    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2z9k\" (UniqueName: \"kubernetes.io/projected/73e1643d-edd2-4464-8a5a-3cb2ab9d8e51-kube-api-access-w2z9k\") pod \"iptables-alerter-r6nbk\" (UID: \"73e1643d-edd2-4464-8a5a-3cb2ab9d8e51\") " pod="openshift-network-operator/iptables-alerter-r6nbk"
Sep 17 16:41:33.718243 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-var-lib-openvswitch\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.718243 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718158    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-ovnkube-config\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.718243 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-device-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.718470 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-sys\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.718470 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.718470 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-ovnkube-script-lib\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.718621 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48abb272-fae7-4755-a74e-f82be2c3b6f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.718621 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n5bb\" (UniqueName: \"kubernetes.io/projected/48abb272-fae7-4755-a74e-f82be2c3b6f0-kube-api-access-9n5bb\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.718621 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-socket-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.718621 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-var-lib-kubelet\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.718621 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b658125-8a32-4475-a8be-5351a5064404-agent-certs\") pod \"konnectivity-agent-264lh\" (UID: \"7b658125-8a32-4475-a8be-5351a5064404\") " pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:33.718819 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-env-overrides\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.718819 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48abb272-fae7-4755-a74e-f82be2c3b6f0-cnibin\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.718819 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-systemd-units\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.718819 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-node-log\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.718819 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df7d88b0-b648-45ed-8381-e7172a404315-dbus\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:33.718819 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe-tmp-dir\") pod \"node-resolver-8hvh7\" (UID: \"55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe\") " pod="openshift-dns/node-resolver-8hvh7"
Sep 17 16:41:33.718819 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48abb272-fae7-4755-a74e-f82be2c3b6f0-system-cni-dir\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.719067 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718835    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.719067 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-run-netns\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.719067 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-var-lib-cni-bin\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.719067 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73e1643d-edd2-4464-8a5a-3cb2ab9d8e51-iptables-alerter-script\") pod \"iptables-alerter-r6nbk\" (UID: \"73e1643d-edd2-4464-8a5a-3cb2ab9d8e51\") " pod="openshift-network-operator/iptables-alerter-r6nbk"
Sep 17 16:41:33.719067 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718925    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73e1643d-edd2-4464-8a5a-3cb2ab9d8e51-host-slash\") pod \"iptables-alerter-r6nbk\" (UID: \"73e1643d-edd2-4464-8a5a-3cb2ab9d8e51\") " pod="openshift-network-operator/iptables-alerter-r6nbk"
Sep 17 16:41:33.719067 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-etc-openvswitch\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.719067 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-sys-fs\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.719067 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.718993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-tuned\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.719067 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-conf-dir\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.719067 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b658125-8a32-4475-a8be-5351a5064404-konnectivity-ca\") pod \"konnectivity-agent-264lh\" (UID: \"7b658125-8a32-4475-a8be-5351a5064404\") " pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-slash\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-run-ovn\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df7d88b0-b648-45ed-8381-e7172a404315-kubelet-config\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719163    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2dkw\" (UniqueName: \"kubernetes.io/projected/55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe-kube-api-access-r2dkw\") pod \"node-resolver-8hvh7\" (UID: \"55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe\") " pod="openshift-dns/node-resolver-8hvh7"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-sysctl-d\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719237    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-lib-modules\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51a83d4f-a400-4730-bc2a-2d42aad6ab99-tmp\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719322    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft45g\" (UniqueName: \"kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g\") pod \"network-check-target-w6lnl\" (UID: \"8a1950af-8604-4048-93b4-1bdb996f291e\") " pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-kubernetes\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-host\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-run-k8s-cni-cncf-io\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-var-lib-cni-multus\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719469    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-etc-kubernetes\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-run-netns\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.719506 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-cni-netd\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-cni-dir\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-var-lib-kubelet\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwcns\" (UniqueName: \"kubernetes.io/projected/4efac150-5db6-42c1-8309-138120e7f820-kube-api-access-xwcns\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-log-socket\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719642    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-registration-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719677    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719700    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48abb272-fae7-4755-a74e-f82be2c3b6f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-hostroot\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719767    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-ovn-node-metrics-cert\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719783    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48abb272-fae7-4755-a74e-f82be2c3b6f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-cnibin\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-run-multus-certs\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-kubelet\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-systemd\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.720008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dce7e7cb-88b0-4419-b2c9-144f94438855-cni-binary-copy\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe-hosts-file\") pod \"node-resolver-8hvh7\" (UID: \"55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe\") " pod="openshift-dns/node-resolver-8hvh7"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-modprobe-d\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-socket-dir-parent\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719968    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-daemon-config\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.719991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10-serviceca\") pod \"node-ca-h774k\" (UID: \"89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10\") " pod="openshift-image-registry/node-ca-h774k"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqw5t\" (UniqueName: \"kubernetes.io/projected/89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10-kube-api-access-jqw5t\") pod \"node-ca-h774k\" (UID: \"89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10\") " pod="openshift-image-registry/node-ca-h774k"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-cni-bin\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-sysconfig\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-sysctl-conf\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-run-openvswitch\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vtx6\" (UniqueName: \"kubernetes.io/projected/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-kube-api-access-9vtx6\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48abb272-fae7-4755-a74e-f82be2c3b6f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnj6q\" (UniqueName: \"kubernetes.io/projected/51a83d4f-a400-4730-bc2a-2d42aad6ab99-kube-api-access-jnj6q\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-run-systemd\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-etc-selinux\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.720628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9pbm\" (UniqueName: \"kubernetes.io/projected/97af9937-e3bc-49b1-9fb2-c250e10418da-kube-api-access-v9pbm\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.721244 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-run\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.721244 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.720355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-system-cni-dir\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.745113 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.745091    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zb2km"
Sep 17 16:41:33.754268 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.753931    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zb2km"
Sep 17 16:41:33.807214 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.807194    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 16:41:33.807655 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.807618    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal" event={"ID":"3ece54a7e142bdf40d1f2f69c78afd8b","Type":"ContainerStarted","Data":"a3a53eae1ea8338dfb02b8c291f0fccea9b3c61e1668f6d88ebc5061a08ff98e"}
Sep 17 16:41:33.808639 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.808619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-202.ec2.internal" event={"ID":"c990e3bd2b767d7e89440c7596b76213","Type":"ContainerStarted","Data":"01057746c71ad58d0a811bfa4e08e497ef7faaac260be71f64e90b2796eb0371"}
Sep 17 16:41:33.821375 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-ovnkube-script-lib\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.821466 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48abb272-fae7-4755-a74e-f82be2c3b6f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.821466 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n5bb\" (UniqueName: \"kubernetes.io/projected/48abb272-fae7-4755-a74e-f82be2c3b6f0-kube-api-access-9n5bb\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.821466 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-socket-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.821466 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-var-lib-kubelet\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.821466 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b658125-8a32-4475-a8be-5351a5064404-agent-certs\") pod \"konnectivity-agent-264lh\" (UID: \"7b658125-8a32-4475-a8be-5351a5064404\") " pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:33.821711 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821488    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-env-overrides\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.821711 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48abb272-fae7-4755-a74e-f82be2c3b6f0-cnibin\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.821711 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-systemd-units\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.821711 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821538    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-socket-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.821711 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821565    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-var-lib-kubelet\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.821711 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-node-log\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.821711 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821603    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-node-log\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.821711 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df7d88b0-b648-45ed-8381-e7172a404315-dbus\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:33.821711 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821646    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48abb272-fae7-4755-a74e-f82be2c3b6f0-cnibin\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.821711 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe-tmp-dir\") pod \"node-resolver-8hvh7\" (UID: \"55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe\") " pod="openshift-dns/node-resolver-8hvh7"
Sep 17 16:41:33.821711 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-systemd-units\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.822114 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48abb272-fae7-4755-a74e-f82be2c3b6f0-system-cni-dir\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.822114 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.822114 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-run-netns\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.822114 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df7d88b0-b648-45ed-8381-e7172a404315-dbus\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:33.822368 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.821781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48abb272-fae7-4755-a74e-f82be2c3b6f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.822512 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-env-overrides\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.822594 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48abb272-fae7-4755-a74e-f82be2c3b6f0-system-cni-dir\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.822640 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822614    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 16:41:33.822640 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.822731 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-run-netns\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.822731 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-var-lib-cni-bin\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.822731 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73e1643d-edd2-4464-8a5a-3cb2ab9d8e51-iptables-alerter-script\") pod \"iptables-alerter-r6nbk\" (UID: \"73e1643d-edd2-4464-8a5a-3cb2ab9d8e51\") " pod="openshift-network-operator/iptables-alerter-r6nbk"
Sep 17 16:41:33.822731 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-var-lib-cni-bin\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.822731 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73e1643d-edd2-4464-8a5a-3cb2ab9d8e51-host-slash\") pod \"iptables-alerter-r6nbk\" (UID: \"73e1643d-edd2-4464-8a5a-3cb2ab9d8e51\") " pod="openshift-network-operator/iptables-alerter-r6nbk"
Sep 17 16:41:33.822731 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-ovnkube-script-lib\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-etc-openvswitch\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822757    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73e1643d-edd2-4464-8a5a-3cb2ab9d8e51-host-slash\") pod \"iptables-alerter-r6nbk\" (UID: \"73e1643d-edd2-4464-8a5a-3cb2ab9d8e51\") " pod="openshift-network-operator/iptables-alerter-r6nbk"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe-tmp-dir\") pod \"node-resolver-8hvh7\" (UID: \"55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe\") " pod="openshift-dns/node-resolver-8hvh7"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-sys-fs\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-tuned\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-sys-fs\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-conf-dir\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-etc-openvswitch\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822860    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-conf-dir\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b658125-8a32-4475-a8be-5351a5064404-konnectivity-ca\") pod \"konnectivity-agent-264lh\" (UID: \"7b658125-8a32-4475-a8be-5351a5064404\") " pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-slash\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822918    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-slash\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-run-ovn\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df7d88b0-b648-45ed-8381-e7172a404315-kubelet-config\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-run-ovn\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.823010 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.822999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2dkw\" (UniqueName: \"kubernetes.io/projected/55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe-kube-api-access-r2dkw\") pod \"node-resolver-8hvh7\" (UID: \"55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe\") " pod="openshift-dns/node-resolver-8hvh7"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-sysctl-d\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-lib-modules\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51a83d4f-a400-4730-bc2a-2d42aad6ab99-tmp\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823086    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft45g\" (UniqueName: \"kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g\") pod \"network-check-target-w6lnl\" (UID: \"8a1950af-8604-4048-93b4-1bdb996f291e\") " pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-kubernetes\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-host\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-run-k8s-cni-cncf-io\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-var-lib-cni-multus\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-lib-modules\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823203    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-etc-kubernetes\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73e1643d-edd2-4464-8a5a-3cb2ab9d8e51-iptables-alerter-script\") pod \"iptables-alerter-r6nbk\" (UID: \"73e1643d-edd2-4464-8a5a-3cb2ab9d8e51\") " pod="openshift-network-operator/iptables-alerter-r6nbk"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-etc-kubernetes\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-run-netns\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df7d88b0-b648-45ed-8381-e7172a404315-kubelet-config\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-cni-netd\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-run-k8s-cni-cncf-io\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.823714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823358    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-sysctl-d\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823333    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-host\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823360    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-run-netns\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-kubernetes\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-cni-dir\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823396    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-cni-netd\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b658125-8a32-4475-a8be-5351a5064404-konnectivity-ca\") pod \"konnectivity-agent-264lh\" (UID: \"7b658125-8a32-4475-a8be-5351a5064404\") " pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823435    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-var-lib-cni-multus\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:33.823495    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-var-lib-kubelet\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:33.823557    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret podName:df7d88b0-b648-45ed-8381-e7172a404315 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:34.323538924 +0000 UTC m=+3.035546192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret") pod "global-pull-secret-syncer-qw97h" (UID: "df7d88b0-b648-45ed-8381-e7172a404315") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-cni-dir\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcns\" (UniqueName: \"kubernetes.io/projected/4efac150-5db6-42c1-8309-138120e7f820-kube-api-access-xwcns\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-var-lib-kubelet\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823631    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-log-socket\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.824582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-registration-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823755    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-log-socket\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-registration-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823852    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48abb272-fae7-4755-a74e-f82be2c3b6f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:33.823903    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-hostroot\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-ovn-node-metrics-cert\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48abb272-fae7-4755-a74e-f82be2c3b6f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.823997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-hostroot\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-cnibin\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-run-multus-certs\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824154    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-kubelet\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-systemd\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824199    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dce7e7cb-88b0-4419-b2c9-144f94438855-cni-binary-copy\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824236    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe-hosts-file\") pod \"node-resolver-8hvh7\" (UID: \"55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe\") " pod="openshift-dns/node-resolver-8hvh7"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-modprobe-d\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-kubelet\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.825413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48abb272-fae7-4755-a74e-f82be2c3b6f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe-hosts-file\") pod \"node-resolver-8hvh7\" (UID: \"55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe\") " pod="openshift-dns/node-resolver-8hvh7"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-systemd\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-modprobe-d\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-socket-dir-parent\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-daemon-config\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10-serviceca\") pod \"node-ca-h774k\" (UID: \"89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10\") " pod="openshift-image-registry/node-ca-h774k"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqw5t\" (UniqueName: \"kubernetes.io/projected/89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10-kube-api-access-jqw5t\") pod \"node-ca-h774k\" (UID: \"89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10\") " pod="openshift-image-registry/node-ca-h774k"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-cni-bin\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-sysconfig\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-sysctl-conf\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-run-openvswitch\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vtx6\" (UniqueName: \"kubernetes.io/projected/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-kube-api-access-9vtx6\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.824951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dce7e7cb-88b0-4419-b2c9-144f94438855-cni-binary-copy\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48abb272-fae7-4755-a74e-f82be2c3b6f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825014    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-run-openvswitch\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnj6q\" (UniqueName: \"kubernetes.io/projected/51a83d4f-a400-4730-bc2a-2d42aad6ab99-kube-api-access-jnj6q\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.826308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48abb272-fae7-4755-a74e-f82be2c3b6f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825129    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-cnibin\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-host-run-multus-certs\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-sysconfig\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:33.825223    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs podName:4efac150-5db6-42c1-8309-138120e7f820 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:34.325207048 +0000 UTC m=+3.037214314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs") pod "network-metrics-daemon-vm27c" (UID: "4efac150-5db6-42c1-8309-138120e7f820") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-cni-bin\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-socket-dir-parent\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-run-systemd\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-sysctl-conf\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825486    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-etc-selinux\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9pbm\" (UniqueName: \"kubernetes.io/projected/97af9937-e3bc-49b1-9fb2-c250e10418da-kube-api-access-v9pbm\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-run\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-system-cni-dir\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5xmc\" (UniqueName: \"kubernetes.io/projected/dce7e7cb-88b0-4419-b2c9-144f94438855-kube-api-access-k5xmc\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825663    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10-host\") pod \"node-ca-h774k\" (UID: \"89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10\") " pod="openshift-image-registry/node-ca-h774k"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2z9k\" (UniqueName: \"kubernetes.io/projected/73e1643d-edd2-4464-8a5a-3cb2ab9d8e51-kube-api-access-w2z9k\") pod \"iptables-alerter-r6nbk\" (UID: \"73e1643d-edd2-4464-8a5a-3cb2ab9d8e51\") " pod="openshift-network-operator/iptables-alerter-r6nbk"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dce7e7cb-88b0-4419-b2c9-144f94438855-multus-daemon-config\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.827149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825755    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-var-lib-openvswitch\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825757    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10-serviceca\") pod \"node-ca-h774k\" (UID: \"89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10\") " pod="openshift-image-registry/node-ca-h774k"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825710    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-var-lib-openvswitch\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825757    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce7e7cb-88b0-4419-b2c9-144f94438855-system-cni-dir\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-run-systemd\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-ovnkube-config\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-device-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-sys\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.825977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.826238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10-host\") pod \"node-ca-h774k\" (UID: \"89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10\") " pod="openshift-image-registry/node-ca-h774k"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.826278    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48abb272-fae7-4755-a74e-f82be2c3b6f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.826330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-run\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.826394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-etc-selinux\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.826581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51a83d4f-a400-4730-bc2a-2d42aad6ab99-etc-tuned\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.826587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97af9937-e3bc-49b1-9fb2-c250e10418da-device-dir\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.826622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51a83d4f-a400-4730-bc2a-2d42aad6ab99-sys\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.826636    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-ovnkube-config\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.827784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.827016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b658125-8a32-4475-a8be-5351a5064404-agent-certs\") pod \"konnectivity-agent-264lh\" (UID: \"7b658125-8a32-4475-a8be-5351a5064404\") " pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:33.828415 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.827122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51a83d4f-a400-4730-bc2a-2d42aad6ab99-tmp\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.828629 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.828614    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-ovn-node-metrics-cert\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.830548 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:33.830171    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:33.830548 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:33.830197    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:33.830548 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:33.830209    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ft45g for pod openshift-network-diagnostics/network-check-target-w6lnl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:33.830548 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:33.830285    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g podName:8a1950af-8604-4048-93b4-1bdb996f291e nodeName:}" failed. No retries permitted until 2026-09-17 16:41:34.330269505 +0000 UTC m=+3.042276776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ft45g" (UniqueName: "kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g") pod "network-check-target-w6lnl" (UID: "8a1950af-8604-4048-93b4-1bdb996f291e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:33.833348 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.832536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2dkw\" (UniqueName: \"kubernetes.io/projected/55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe-kube-api-access-r2dkw\") pod \"node-resolver-8hvh7\" (UID: \"55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe\") " pod="openshift-dns/node-resolver-8hvh7"
Sep 17 16:41:33.834035 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.834017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwcns\" (UniqueName: \"kubernetes.io/projected/4efac150-5db6-42c1-8309-138120e7f820-kube-api-access-xwcns\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:33.835764 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.834892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n5bb\" (UniqueName: \"kubernetes.io/projected/48abb272-fae7-4755-a74e-f82be2c3b6f0-kube-api-access-9n5bb\") pod \"multus-additional-cni-plugins-lkvgc\" (UID: \"48abb272-fae7-4755-a74e-f82be2c3b6f0\") " pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:33.836006 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.835969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vtx6\" (UniqueName: \"kubernetes.io/projected/c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b-kube-api-access-9vtx6\") pod \"ovnkube-node-cxtcx\" (UID: \"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:33.836524 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.836088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqw5t\" (UniqueName: \"kubernetes.io/projected/89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10-kube-api-access-jqw5t\") pod \"node-ca-h774k\" (UID: \"89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10\") " pod="openshift-image-registry/node-ca-h774k"
Sep 17 16:41:33.836524 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.836414    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9pbm\" (UniqueName: \"kubernetes.io/projected/97af9937-e3bc-49b1-9fb2-c250e10418da-kube-api-access-v9pbm\") pod \"aws-ebs-csi-driver-node-74rdx\" (UID: \"97af9937-e3bc-49b1-9fb2-c250e10418da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:33.836897 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.836878    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnj6q\" (UniqueName: \"kubernetes.io/projected/51a83d4f-a400-4730-bc2a-2d42aad6ab99-kube-api-access-jnj6q\") pod \"tuned-82b45\" (UID: \"51a83d4f-a400-4730-bc2a-2d42aad6ab99\") " pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:33.837083 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.837067    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5xmc\" (UniqueName: \"kubernetes.io/projected/dce7e7cb-88b0-4419-b2c9-144f94438855-kube-api-access-k5xmc\") pod \"multus-x4282\" (UID: \"dce7e7cb-88b0-4419-b2c9-144f94438855\") " pod="openshift-multus/multus-x4282"
Sep 17 16:41:33.837960 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.837945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2z9k\" (UniqueName: \"kubernetes.io/projected/73e1643d-edd2-4464-8a5a-3cb2ab9d8e51-kube-api-access-w2z9k\") pod \"iptables-alerter-r6nbk\" (UID: \"73e1643d-edd2-4464-8a5a-3cb2ab9d8e51\") " pod="openshift-network-operator/iptables-alerter-r6nbk"
Sep 17 16:41:33.995684 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:33.995597    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:34.004394 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.004370    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8hvh7"
Sep 17 16:41:34.016015 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.015985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h774k"
Sep 17 16:41:34.021590 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.021570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lkvgc"
Sep 17 16:41:34.030303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.030283    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx"
Sep 17 16:41:34.037226 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.037205    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-82b45"
Sep 17 16:41:34.037462 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.037441    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:34.045801 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.045781    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x4282"
Sep 17 16:41:34.054239 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.054221    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6nbk"
Sep 17 16:41:34.060933 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.060914    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:34.328871 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.328786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:34.328871 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.328859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:34.329085 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:34.328908    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:34.329085 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:34.328978    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:34.329085 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:34.328996    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret podName:df7d88b0-b648-45ed-8381-e7172a404315 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:35.328976853 +0000 UTC m=+4.040984116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret") pod "global-pull-secret-syncer-qw97h" (UID: "df7d88b0-b648-45ed-8381-e7172a404315") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:34.329085 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:34.329027    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs podName:4efac150-5db6-42c1-8309-138120e7f820 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:35.329011713 +0000 UTC m=+4.041018965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs") pod "network-metrics-daemon-vm27c" (UID: "4efac150-5db6-42c1-8309-138120e7f820") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:34.344114 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.344092    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:34.429329 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.429297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft45g\" (UniqueName: \"kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g\") pod \"network-check-target-w6lnl\" (UID: \"8a1950af-8604-4048-93b4-1bdb996f291e\") " pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:34.429492 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:34.429472    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:34.429558 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:34.429503    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:34.429558 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:34.429517    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ft45g for pod openshift-network-diagnostics/network-check-target-w6lnl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:34.429659 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:34.429579    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g podName:8a1950af-8604-4048-93b4-1bdb996f291e nodeName:}" failed. No retries permitted until 2026-09-17 16:41:35.429558583 +0000 UTC m=+4.141565843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft45g" (UniqueName: "kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g") pod "network-check-target-w6lnl" (UID: "8a1950af-8604-4048-93b4-1bdb996f291e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:34.757336 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.757248    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:36:33 +0000 UTC" deadline="2028-04-06 16:00:57.928665209 +0000 UTC"
Sep 17 16:41:34.757336 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.757292    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13607h19m23.171376348s"
Sep 17 16:41:34.804709 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:34.804680    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:34.804856 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:34.804832    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:34.819533 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:34.819503    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a83d4f_a400_4730_bc2a_2d42aad6ab99.slice/crio-bc789613c2d84ed0a14e44e4a7c76568ce29d5272512c1028db7708d8faf05b2 WatchSource:0}: Error finding container bc789613c2d84ed0a14e44e4a7c76568ce29d5272512c1028db7708d8faf05b2: Status 404 returned error can't find the container with id bc789613c2d84ed0a14e44e4a7c76568ce29d5272512c1028db7708d8faf05b2
Sep 17 16:41:34.822597 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:34.822430    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48abb272_fae7_4755_a74e_f82be2c3b6f0.slice/crio-0599177dd8190b55c831e047defebde8b29255ebd253edcf67f7af79c1df4acc WatchSource:0}: Error finding container 0599177dd8190b55c831e047defebde8b29255ebd253edcf67f7af79c1df4acc: Status 404 returned error can't find the container with id 0599177dd8190b55c831e047defebde8b29255ebd253edcf67f7af79c1df4acc
Sep 17 16:41:34.825591 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:34.825567    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a6b9e3_1b73_4a0e_b75d_1c4c8e81314b.slice/crio-8e52c1d8418ed171ad37334d8cb85de757bd56ecc9422c7d70aad3b8a05a2c15 WatchSource:0}: Error finding container 8e52c1d8418ed171ad37334d8cb85de757bd56ecc9422c7d70aad3b8a05a2c15: Status 404 returned error can't find the container with id 8e52c1d8418ed171ad37334d8cb85de757bd56ecc9422c7d70aad3b8a05a2c15
Sep 17 16:41:34.826442 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:34.826419    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55dd2af7_8fe9_4f9d_aad5_680fdeb3dfbe.slice/crio-548b149120f0c7c161e11532b57f920eced05661d574b7cc651b86076c7bf04a WatchSource:0}: Error finding container 548b149120f0c7c161e11532b57f920eced05661d574b7cc651b86076c7bf04a: Status 404 returned error can't find the container with id 548b149120f0c7c161e11532b57f920eced05661d574b7cc651b86076c7bf04a
Sep 17 16:41:34.827327 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:34.827228    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce7e7cb_88b0_4419_b2c9_144f94438855.slice/crio-c4518e258aee91690972f10fa0f7db4d4f3bfdafd563de9304fa80c34f1c3837 WatchSource:0}: Error finding container c4518e258aee91690972f10fa0f7db4d4f3bfdafd563de9304fa80c34f1c3837: Status 404 returned error can't find the container with id c4518e258aee91690972f10fa0f7db4d4f3bfdafd563de9304fa80c34f1c3837
Sep 17 16:41:34.828299 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:34.828269    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e1643d_edd2_4464_8a5a_3cb2ab9d8e51.slice/crio-febabad8c7b33fb7adbdd5f297aa8495b858b2a28ab122e2107f89680a2ecc00 WatchSource:0}: Error finding container febabad8c7b33fb7adbdd5f297aa8495b858b2a28ab122e2107f89680a2ecc00: Status 404 returned error can't find the container with id febabad8c7b33fb7adbdd5f297aa8495b858b2a28ab122e2107f89680a2ecc00
Sep 17 16:41:34.829573 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:34.829507    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d257ca_2fd9_45a8_b0a3_7cfff6e8cd10.slice/crio-205c937bf222feb92d8e0a0a317b95668505d648ad1e9924cd39cd5fc3ee4b0a WatchSource:0}: Error finding container 205c937bf222feb92d8e0a0a317b95668505d648ad1e9924cd39cd5fc3ee4b0a: Status 404 returned error can't find the container with id 205c937bf222feb92d8e0a0a317b95668505d648ad1e9924cd39cd5fc3ee4b0a
Sep 17 16:41:34.831481 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:34.831436    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b658125_8a32_4475_a8be_5351a5064404.slice/crio-1f648c54b1af3087055d9bb9581d5f1af66eb55de522e23bde9a4f4dacf36208 WatchSource:0}: Error finding container 1f648c54b1af3087055d9bb9581d5f1af66eb55de522e23bde9a4f4dacf36208: Status 404 returned error can't find the container with id 1f648c54b1af3087055d9bb9581d5f1af66eb55de522e23bde9a4f4dacf36208
Sep 17 16:41:34.833531 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:41:34.833497    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97af9937_e3bc_49b1_9fb2_c250e10418da.slice/crio-61915e626c115aff2690706b0451efc360854461249106d0e14d43da591635d1 WatchSource:0}: Error finding container 61915e626c115aff2690706b0451efc360854461249106d0e14d43da591635d1: Status 404 returned error can't find the container with id 61915e626c115aff2690706b0451efc360854461249106d0e14d43da591635d1
Sep 17 16:41:35.337403 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.336832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:35.337403 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.337117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:35.337403 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:35.337150    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:35.337403 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:35.337218    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret podName:df7d88b0-b648-45ed-8381-e7172a404315 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:37.337198006 +0000 UTC m=+6.049205258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret") pod "global-pull-secret-syncer-qw97h" (UID: "df7d88b0-b648-45ed-8381-e7172a404315") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:35.337403 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:35.337241    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:35.337403 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:35.337325    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs podName:4efac150-5db6-42c1-8309-138120e7f820 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:37.337308304 +0000 UTC m=+6.049315554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs") pod "network-metrics-daemon-vm27c" (UID: "4efac150-5db6-42c1-8309-138120e7f820") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:35.437969 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.437437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft45g\" (UniqueName: \"kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g\") pod \"network-check-target-w6lnl\" (UID: \"8a1950af-8604-4048-93b4-1bdb996f291e\") " pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:35.437969 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:35.437579    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:35.437969 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:35.437602    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:35.437969 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:35.437615    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ft45g for pod openshift-network-diagnostics/network-check-target-w6lnl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:35.437969 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:35.437666    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g podName:8a1950af-8604-4048-93b4-1bdb996f291e nodeName:}" failed. No retries permitted until 2026-09-17 16:41:37.43764989 +0000 UTC m=+6.149657139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft45g" (UniqueName: "kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g") pod "network-check-target-w6lnl" (UID: "8a1950af-8604-4048-93b4-1bdb996f291e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:35.757712 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.757605    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:36:33 +0000 UTC" deadline="2028-06-01 15:56:37.443643324 +0000 UTC"
Sep 17 16:41:35.757712 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.757667    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14951h15m1.685980274s"
Sep 17 16:41:35.805861 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.805359    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:35.805861 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:35.805476    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:35.806087 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.805982    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:35.806138 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:35.806086    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:35.838261 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.838185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h774k" event={"ID":"89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10","Type":"ContainerStarted","Data":"205c937bf222feb92d8e0a0a317b95668505d648ad1e9924cd39cd5fc3ee4b0a"}
Sep 17 16:41:35.844980 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.844949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6nbk" event={"ID":"73e1643d-edd2-4464-8a5a-3cb2ab9d8e51","Type":"ContainerStarted","Data":"febabad8c7b33fb7adbdd5f297aa8495b858b2a28ab122e2107f89680a2ecc00"}
Sep 17 16:41:35.859578 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.859503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x4282" event={"ID":"dce7e7cb-88b0-4419-b2c9-144f94438855","Type":"ContainerStarted","Data":"c4518e258aee91690972f10fa0f7db4d4f3bfdafd563de9304fa80c34f1c3837"}
Sep 17 16:41:35.866081 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.866057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-202.ec2.internal" event={"ID":"c990e3bd2b767d7e89440c7596b76213","Type":"ContainerStarted","Data":"3bf41a7b7f632155ac0b096300c378762fd8a8cb51e164a91465b72b5da86893"}
Sep 17 16:41:35.893855 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.893830    2566 generic.go:356] "Generic (PLEG): container finished" podID="3ece54a7e142bdf40d1f2f69c78afd8b" containerID="0cef232a8010be64850959c6fbb08c89cc80f4cf5a1ac931680a269f1496546b" exitCode=0
Sep 17 16:41:35.893966 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.893896    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal" event={"ID":"3ece54a7e142bdf40d1f2f69c78afd8b","Type":"ContainerDied","Data":"0cef232a8010be64850959c6fbb08c89cc80f4cf5a1ac931680a269f1496546b"}
Sep 17 16:41:35.910121 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.908880    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx" event={"ID":"97af9937-e3bc-49b1-9fb2-c250e10418da","Type":"ContainerStarted","Data":"61915e626c115aff2690706b0451efc360854461249106d0e14d43da591635d1"}
Sep 17 16:41:35.910121 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.909183    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-202.ec2.internal" podStartSLOduration=3.909169167 podStartE2EDuration="3.909169167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:35.881525479 +0000 UTC m=+4.593532750" watchObservedRunningTime="2026-09-17 16:41:35.909169167 +0000 UTC m=+4.621176436"
Sep 17 16:41:35.921143 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.921058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8hvh7" event={"ID":"55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe","Type":"ContainerStarted","Data":"548b149120f0c7c161e11532b57f920eced05661d574b7cc651b86076c7bf04a"}
Sep 17 16:41:35.935695 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.935669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx" event={"ID":"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b","Type":"ContainerStarted","Data":"8e52c1d8418ed171ad37334d8cb85de757bd56ecc9422c7d70aad3b8a05a2c15"}
Sep 17 16:41:35.943807 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.943782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkvgc" event={"ID":"48abb272-fae7-4755-a74e-f82be2c3b6f0","Type":"ContainerStarted","Data":"0599177dd8190b55c831e047defebde8b29255ebd253edcf67f7af79c1df4acc"}
Sep 17 16:41:35.948419 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.948393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-82b45" event={"ID":"51a83d4f-a400-4730-bc2a-2d42aad6ab99","Type":"ContainerStarted","Data":"bc789613c2d84ed0a14e44e4a7c76568ce29d5272512c1028db7708d8faf05b2"}
Sep 17 16:41:35.953946 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:35.953922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-264lh" event={"ID":"7b658125-8a32-4475-a8be-5351a5064404","Type":"ContainerStarted","Data":"1f648c54b1af3087055d9bb9581d5f1af66eb55de522e23bde9a4f4dacf36208"}
Sep 17 16:41:36.803934 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:36.803904    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:36.804403 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:36.804030    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:36.975813 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:36.975683    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal" event={"ID":"3ece54a7e142bdf40d1f2f69c78afd8b","Type":"ContainerStarted","Data":"3bf1ddc97308339fcea4636b00c74c99de472753d2b88a20a276a9180f911d10"}
Sep 17 16:41:36.992014 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:36.991026    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-202.ec2.internal" podStartSLOduration=4.99100627 podStartE2EDuration="4.99100627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:36.989986628 +0000 UTC m=+5.701993895" watchObservedRunningTime="2026-09-17 16:41:36.99100627 +0000 UTC m=+5.703013541"
Sep 17 16:41:37.357576 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:37.356761    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:37.357576 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:37.356850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:37.357576 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:37.356986    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:37.357576 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:37.357043    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret podName:df7d88b0-b648-45ed-8381-e7172a404315 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:41.357025583 +0000 UTC m=+10.069032843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret") pod "global-pull-secret-syncer-qw97h" (UID: "df7d88b0-b648-45ed-8381-e7172a404315") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:37.357576 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:37.357438    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:37.357576 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:37.357487    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs podName:4efac150-5db6-42c1-8309-138120e7f820 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:41.357472774 +0000 UTC m=+10.069480023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs") pod "network-metrics-daemon-vm27c" (UID: "4efac150-5db6-42c1-8309-138120e7f820") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:37.457709 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:37.457140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft45g\" (UniqueName: \"kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g\") pod \"network-check-target-w6lnl\" (UID: \"8a1950af-8604-4048-93b4-1bdb996f291e\") " pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:37.457709 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:37.457317    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:37.457709 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:37.457337    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:37.457709 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:37.457350    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ft45g for pod openshift-network-diagnostics/network-check-target-w6lnl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:37.457709 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:37.457402    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g podName:8a1950af-8604-4048-93b4-1bdb996f291e nodeName:}" failed. No retries permitted until 2026-09-17 16:41:41.45738534 +0000 UTC m=+10.169392601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft45g" (UniqueName: "kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g") pod "network-check-target-w6lnl" (UID: "8a1950af-8604-4048-93b4-1bdb996f291e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:37.804407 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:37.804336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:37.804827 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:37.804459    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:37.805975 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:37.805948    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:37.806094 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:37.806047    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:38.804595 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:38.804562    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:38.805051 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:38.804710    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:39.804796 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:39.804762    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:39.805228 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:39.804769    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:39.805228 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:39.804888    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:39.805228 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:39.804994    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:40.804587 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:40.804557    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:40.804765 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:40.804687    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:41.388625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:41.388597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:41.389076 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:41.388653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:41.389076 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:41.388752    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:41.389076 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:41.388809    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs podName:4efac150-5db6-42c1-8309-138120e7f820 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:49.388792075 +0000 UTC m=+18.100799321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs") pod "network-metrics-daemon-vm27c" (UID: "4efac150-5db6-42c1-8309-138120e7f820") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:41.389076 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:41.388752    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:41.389076 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:41.388873    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret podName:df7d88b0-b648-45ed-8381-e7172a404315 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:49.388859547 +0000 UTC m=+18.100866806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret") pod "global-pull-secret-syncer-qw97h" (UID: "df7d88b0-b648-45ed-8381-e7172a404315") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:41.490275 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:41.489826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft45g\" (UniqueName: \"kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g\") pod \"network-check-target-w6lnl\" (UID: \"8a1950af-8604-4048-93b4-1bdb996f291e\") " pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:41.490275 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:41.489991    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:41.490275 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:41.490009    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:41.490275 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:41.490021    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ft45g for pod openshift-network-diagnostics/network-check-target-w6lnl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:41.490275 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:41.490071    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g podName:8a1950af-8604-4048-93b4-1bdb996f291e nodeName:}" failed. No retries permitted until 2026-09-17 16:41:49.490055306 +0000 UTC m=+18.202062552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft45g" (UniqueName: "kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g") pod "network-check-target-w6lnl" (UID: "8a1950af-8604-4048-93b4-1bdb996f291e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:41.804706 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:41.804627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:41.804856 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:41.804729    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:41.804856 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:41.804803    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:41.804950 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:41.804922    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:42.804396 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:42.804311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:42.804817 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:42.804454    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:43.804072 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:43.804039    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:43.804231 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:43.804086    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:43.804231 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:43.804176    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:43.804391 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:43.804358    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:44.804281 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:44.804243    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:44.804715 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:44.804389    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:45.804179 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:45.804144    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:45.804179 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:45.804185    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:45.804703 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:45.804288    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:45.804703 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:45.804422    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:46.804305 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:46.804276    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:46.804454 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:46.804404    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:47.804192 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:47.804149    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:47.804357 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:47.804161    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:47.804357 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:47.804287    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:47.804472 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:47.804362    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:48.804675 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:48.804605    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:48.805162 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:48.804728    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:49.447568 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:49.447533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:49.447744 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:49.447587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:49.447744 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:49.447690    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:49.447744 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:49.447696    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:49.447744 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:49.447746    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs podName:4efac150-5db6-42c1-8309-138120e7f820 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:05.447727757 +0000 UTC m=+34.159735002 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs") pod "network-metrics-daemon-vm27c" (UID: "4efac150-5db6-42c1-8309-138120e7f820") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:49.447910 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:49.447766    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret podName:df7d88b0-b648-45ed-8381-e7172a404315 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:05.447755514 +0000 UTC m=+34.159762759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret") pod "global-pull-secret-syncer-qw97h" (UID: "df7d88b0-b648-45ed-8381-e7172a404315") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:49.548107 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:49.548078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft45g\" (UniqueName: \"kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g\") pod \"network-check-target-w6lnl\" (UID: \"8a1950af-8604-4048-93b4-1bdb996f291e\") " pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:49.548271 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:49.548206    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:49.548271 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:49.548220    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:49.548271 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:49.548229    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ft45g for pod openshift-network-diagnostics/network-check-target-w6lnl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:49.548397 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:49.548289    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g podName:8a1950af-8604-4048-93b4-1bdb996f291e nodeName:}" failed. No retries permitted until 2026-09-17 16:42:05.548273877 +0000 UTC m=+34.260281124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft45g" (UniqueName: "kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g") pod "network-check-target-w6lnl" (UID: "8a1950af-8604-4048-93b4-1bdb996f291e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:49.804679 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:49.804599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:49.805061 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:49.804611    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:49.805061 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:49.804736    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:49.805061 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:49.804841    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:50.804418 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:50.804385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:50.804591 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:50.804518    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:51.804846 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:51.804808    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:51.805339 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:51.804907    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:51.805339 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:51.804972    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:51.805339 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:51.805045    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:52.804188 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:52.803904    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:52.804296 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:52.804279    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:53.007994 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.007952    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x4282" event={"ID":"dce7e7cb-88b0-4419-b2c9-144f94438855","Type":"ContainerStarted","Data":"2cdca8db1db3bde6a713cd41e9eee589504248b926062339dbc2a1cfdefdc247"}
Sep 17 16:41:53.009713 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.009682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx" event={"ID":"97af9937-e3bc-49b1-9fb2-c250e10418da","Type":"ContainerStarted","Data":"b7584104350cb4e758a42cfe6532aef2329d2672e315320467e6e8d750d8d4df"}
Sep 17 16:41:53.011478 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.011451    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8hvh7" event={"ID":"55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe","Type":"ContainerStarted","Data":"ffa7758512c8db8723d53bc6acc442088b66674a02905bd076c3e0fc2829e3c0"}
Sep 17 16:41:53.015885 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.015860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx" event={"ID":"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b","Type":"ContainerStarted","Data":"090e7d54d6261a59db35ad838b089f2dd2d2e791bbd14adeb27f9e3aa49d6e8d"}
Sep 17 16:41:53.015992 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.015932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx" event={"ID":"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b","Type":"ContainerStarted","Data":"4cafa80ce16426661ad2f05931f24ad51869732da4abd20a315ae289b516bd44"}
Sep 17 16:41:53.015992 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.015962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx" event={"ID":"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b","Type":"ContainerStarted","Data":"c19587ee2e902d45b87a9245bc636a74e2d1534ea585963d7ee0a39a2f8448fe"}
Sep 17 16:41:53.015992 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.015989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx" event={"ID":"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b","Type":"ContainerStarted","Data":"afb2a87fb61ec912c9c816f7b2fc979aef6555610cf0d60eb9f8285415fe06f0"}
Sep 17 16:41:53.016153 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.016025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx" event={"ID":"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b","Type":"ContainerStarted","Data":"6b343658c88af9e191cb30c3b8cc3e978f3cf20066380682cd20c286a103e6a8"}
Sep 17 16:41:53.016153 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.016073    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx" event={"ID":"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b","Type":"ContainerStarted","Data":"c17c74b1d20c434b3d05e8b35c1950c02798a0b25b83d7707e062b64f2beac2b"}
Sep 17 16:41:53.017633 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.017511    2566 generic.go:356] "Generic (PLEG): container finished" podID="48abb272-fae7-4755-a74e-f82be2c3b6f0" containerID="a676ca7c2933140420e9bf5b279c27fe4e9047e19d6804406d1a967e317c995f" exitCode=0
Sep 17 16:41:53.017633 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.017593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkvgc" event={"ID":"48abb272-fae7-4755-a74e-f82be2c3b6f0","Type":"ContainerDied","Data":"a676ca7c2933140420e9bf5b279c27fe4e9047e19d6804406d1a967e317c995f"}
Sep 17 16:41:53.019568 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.019530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-82b45" event={"ID":"51a83d4f-a400-4730-bc2a-2d42aad6ab99","Type":"ContainerStarted","Data":"cff3427a7f769e4bf0258dd577ab6c73aff792510b65cae0592884307dd8736c"}
Sep 17 16:41:53.021307 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.021284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-264lh" event={"ID":"7b658125-8a32-4475-a8be-5351a5064404","Type":"ContainerStarted","Data":"574d9434d4c6591644f0c3a567ea0668090a12177ba72420096758693c33ce3d"}
Sep 17 16:41:53.022920 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.022877    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h774k" event={"ID":"89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10","Type":"ContainerStarted","Data":"3618b3cfdb0073d18b1931b5a5e6038caf3ce075147194b0f06d58ba53e9b371"}
Sep 17 16:41:53.025426 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.025388    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x4282" podStartSLOduration=3.421250783 podStartE2EDuration="21.025376462s" totalImagesPullingTime="17.604125679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:53.025170117 +0000 UTC m=+21.737177385" watchObservedRunningTime="2026-09-17 16:41:53.025376462 +0000 UTC m=+21.737383729"
Sep 17 16:41:53.039315 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.039237    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-264lh" podStartSLOduration=7.555974278 podStartE2EDuration="21.039220972s" totalImagesPullingTime="13.483246694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:53.039052777 +0000 UTC m=+21.751060051" watchObservedRunningTime="2026-09-17 16:41:53.039220972 +0000 UTC m=+21.751228261"
Sep 17 16:41:53.080415 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.080375    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-82b45" podStartSLOduration=3.622614579 podStartE2EDuration="21.080363473s" totalImagesPullingTime="17.457748894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:53.079969418 +0000 UTC m=+21.791976684" watchObservedRunningTime="2026-09-17 16:41:53.080363473 +0000 UTC m=+21.792370740"
Sep 17 16:41:53.095856 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.095813    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8hvh7" podStartSLOduration=3.644086493 podStartE2EDuration="21.095800729s" totalImagesPullingTime="17.451714236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:53.095455717 +0000 UTC m=+21.807462985" watchObservedRunningTime="2026-09-17 16:41:53.095800729 +0000 UTC m=+21.807807994"
Sep 17 16:41:53.109326 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.109289    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h774k" podStartSLOduration=3.662092302 podStartE2EDuration="21.109276349s" totalImagesPullingTime="17.447184047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:53.109095336 +0000 UTC m=+21.821102603" watchObservedRunningTime="2026-09-17 16:41:53.109276349 +0000 UTC m=+21.821283621"
Sep 17 16:41:53.767345 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.767314    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 16:41:53.803924 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.803903    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:53.804025 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:53.804008    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:53.804084 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.804066    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:53.804156 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:53.804139    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:53.915188 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.915066    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T16:41:53.767338544Z","UUID":"b041798e-bb49-4cfe-bd0c-27803fc8fd54","Handler":null,"Name":"","Endpoint":""}
Sep 17 16:41:53.916372 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.916351    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 16:41:53.916465 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:53.916383    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 16:41:54.027402 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:54.027369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx" event={"ID":"97af9937-e3bc-49b1-9fb2-c250e10418da","Type":"ContainerStarted","Data":"ba7247b12a87e0e6346d5ef76dab2cf768122ffcdcd6cb8fa8252c86f0923d88"}
Sep 17 16:41:54.029007 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:54.028976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6nbk" event={"ID":"73e1643d-edd2-4464-8a5a-3cb2ab9d8e51","Type":"ContainerStarted","Data":"2810739642e79cd53afbb43ea70f3fb9d57056b7d3ec8c23d691edb775a1dbde"}
Sep 17 16:41:54.045570 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:54.045526    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r6nbk" podStartSLOduration=4.595264811 podStartE2EDuration="22.045511151s" totalImagesPullingTime="17.45024634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:54.044993948 +0000 UTC m=+22.757001220" watchObservedRunningTime="2026-09-17 16:41:54.045511151 +0000 UTC m=+22.757518455"
Sep 17 16:41:54.639271 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:54.639233    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:54.639833 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:54.639811    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:54.804618 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:54.804594    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:54.804766 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:54.804684    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:55.031859 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:55.031831    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:55.032297 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:55.032168    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-264lh"
Sep 17 16:41:55.804615 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:55.804582    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:55.804838 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:55.804715    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:55.804838 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:55.804763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:55.804925 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:55.804843    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:55.980103 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:55.980063    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:41:56.035303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:56.035270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx" event={"ID":"97af9937-e3bc-49b1-9fb2-c250e10418da","Type":"ContainerStarted","Data":"f2e1ea1966e10bb2139fceb46f708540939ecb53cb19c1605c533b5948dfe96a"}
Sep 17 16:41:56.038598 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:56.038573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx" event={"ID":"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b","Type":"ContainerStarted","Data":"cebce9643c41306a88684641589bab6508887cc93e311845d47f36afb07f02ef"}
Sep 17 16:41:56.054265 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:56.054225    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-74rdx" podStartSLOduration=4.283288198 podStartE2EDuration="24.054213348s" totalImagesPullingTime="19.77092515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:56.054091333 +0000 UTC m=+24.766098592" watchObservedRunningTime="2026-09-17 16:41:56.054213348 +0000 UTC m=+24.766220616"
Sep 17 16:41:56.068983 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:56.068924    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:41:56.804788 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:56.804625    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:56.804930 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:56.804863    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:57.804703 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:57.804670    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:57.805614 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:57.804798    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:57.805614 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:57.804868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:57.805614 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:57.804984    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:41:58.047401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:58.047367    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx" event={"ID":"c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b","Type":"ContainerStarted","Data":"82c3182089e13b2d7e5264bb3201e3ee6f8da5ce753e30eaee1b50c203e158f7"}
Sep 17 16:41:58.047684 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:58.047659    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:58.047819 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:58.047748    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:58.064374 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:58.064315    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:58.075242 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:58.075191    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx" podStartSLOduration=8.605475396 podStartE2EDuration="26.075177456s" totalImagesPullingTime="17.46970206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:58.074886361 +0000 UTC m=+26.786893640" watchObservedRunningTime="2026-09-17 16:41:58.075177456 +0000 UTC m=+26.787184724"
Sep 17 16:41:58.804247 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:58.804220    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:58.804430 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:58.804406    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:59.050195 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:59.050016    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:59.076803 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:59.076786    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:41:59.098529 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:59.098428    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qw97h"]
Sep 17 16:41:59.098529 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:59.098503    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:41:59.098650 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:59.098580    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:41:59.105164 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:59.105129    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vm27c"]
Sep 17 16:41:59.105358 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:59.105338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:41:59.105525 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:59.105472    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:41:59.105835 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:59.105816    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w6lnl"]
Sep 17 16:41:59.105930 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:41:59.105894    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:41:59.105995 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:41:59.105975    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:42:00.053347 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:00.053306    2566 generic.go:356] "Generic (PLEG): container finished" podID="48abb272-fae7-4755-a74e-f82be2c3b6f0" containerID="47f64ea6af77b5cc937554687d2fd8c15ee498683dec993c48a07e965f695377" exitCode=0
Sep 17 16:42:00.053872 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:00.053390    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkvgc" event={"ID":"48abb272-fae7-4755-a74e-f82be2c3b6f0","Type":"ContainerDied","Data":"47f64ea6af77b5cc937554687d2fd8c15ee498683dec993c48a07e965f695377"}
Sep 17 16:42:00.804048 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:00.804018    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:42:00.804222 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:00.804018    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:42:00.804222 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:00.804142    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:42:00.804327 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:00.804023    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:42:00.804327 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:00.804225    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:42:00.804405 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:00.804330    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:42:01.325283 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:01.325113    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:01.375909 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:01.375876    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:02.059308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:02.059124    2566 generic.go:356] "Generic (PLEG): container finished" podID="48abb272-fae7-4755-a74e-f82be2c3b6f0" containerID="284b0543a4ff124ca8cd13ffa0fb235c9f8fe4858e9664f717673219e9e12ea2" exitCode=0
Sep 17 16:42:02.059308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:02.059213    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkvgc" event={"ID":"48abb272-fae7-4755-a74e-f82be2c3b6f0","Type":"ContainerDied","Data":"284b0543a4ff124ca8cd13ffa0fb235c9f8fe4858e9664f717673219e9e12ea2"}
Sep 17 16:42:02.774647 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:02.774610    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:02.804283 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:02.804241    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:42:02.804408 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:02.804356    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:42:02.804408 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:02.804380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:42:02.804478 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:02.804444    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:42:02.804478 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:02.804361    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:42:02.804539 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:02.804506    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:42:02.873716 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:02.873690    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:04.065942 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:04.065905    2566 generic.go:356] "Generic (PLEG): container finished" podID="48abb272-fae7-4755-a74e-f82be2c3b6f0" containerID="d0196162999d97cab12150f19d6b035e114abcb69d5a407d21554220f1f734fb" exitCode=0
Sep 17 16:42:04.066362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:04.065966    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkvgc" event={"ID":"48abb272-fae7-4755-a74e-f82be2c3b6f0","Type":"ContainerDied","Data":"d0196162999d97cab12150f19d6b035e114abcb69d5a407d21554220f1f734fb"}
Sep 17 16:42:04.804625 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:04.804573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:42:04.804798 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:04.804573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:42:04.804798 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:04.804720    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qw97h" podUID="df7d88b0-b648-45ed-8381-e7172a404315"
Sep 17 16:42:04.804798 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:04.804573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:42:04.804941 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:04.804792    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vm27c" podUID="4efac150-5db6-42c1-8309-138120e7f820"
Sep 17 16:42:04.804941 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:04.804896    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6lnl" podUID="8a1950af-8604-4048-93b4-1bdb996f291e"
Sep 17 16:42:05.466361 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:05.466280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:42:05.466361 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:05.466333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:42:05.466955 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:05.466428    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:05.466955 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:05.466450    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:05.466955 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:05.466485    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs podName:4efac150-5db6-42c1-8309-138120e7f820 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.466466603 +0000 UTC m=+66.178473850 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs") pod "network-metrics-daemon-vm27c" (UID: "4efac150-5db6-42c1-8309-138120e7f820") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:05.466955 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:05.466508    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret podName:df7d88b0-b648-45ed-8381-e7172a404315 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.466491548 +0000 UTC m=+66.178498799 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret") pod "global-pull-secret-syncer-qw97h" (UID: "df7d88b0-b648-45ed-8381-e7172a404315") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:05.566978 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:05.566951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft45g\" (UniqueName: \"kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g\") pod \"network-check-target-w6lnl\" (UID: \"8a1950af-8604-4048-93b4-1bdb996f291e\") " pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:42:05.567141 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:05.567120    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:05.567205 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:05.567148    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:05.567205 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:05.567162    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ft45g for pod openshift-network-diagnostics/network-check-target-w6lnl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:05.567326 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:05.567215    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g podName:8a1950af-8604-4048-93b4-1bdb996f291e nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.567199128 +0000 UTC m=+66.279206380 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft45g" (UniqueName: "kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g") pod "network-check-target-w6lnl" (UID: "8a1950af-8604-4048-93b4-1bdb996f291e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:06.049551 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.049527    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-202.ec2.internal" event="NodeReady"
Sep 17 16:42:06.049695 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.049633    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 16:42:06.092209 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.092183    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9kh8p"]
Sep 17 16:42:06.095427 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.095141    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.095846 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.095660    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-md9qz"]
Sep 17 16:42:06.097650 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.097630    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 16:42:06.097739 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.097696    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 16:42:06.098030 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.098007    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xc8sp\""
Sep 17 16:42:06.098279 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.098217    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-md9qz"
Sep 17 16:42:06.101059 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.100882    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 16:42:06.101059 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.100960    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 16:42:06.101210 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.101124    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jlx7p\""
Sep 17 16:42:06.101299 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.101277    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 16:42:06.106413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.106394    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9kh8p"]
Sep 17 16:42:06.109030 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.108985    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-md9qz"]
Sep 17 16:42:06.219620 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.219591    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:06.271376 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.271349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde1bd0b-5754-49fd-84c1-28f2383539aa-tmp-dir\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.271511 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.271381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmttr\" (UniqueName: \"kubernetes.io/projected/fde1bd0b-5754-49fd-84c1-28f2383539aa-kube-api-access-hmttr\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.271511 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.271420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b-cert\") pod \"ingress-canary-md9qz\" (UID: \"be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b\") " pod="openshift-ingress-canary/ingress-canary-md9qz"
Sep 17 16:42:06.271622 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.271523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwtbb\" (UniqueName: \"kubernetes.io/projected/be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b-kube-api-access-mwtbb\") pod \"ingress-canary-md9qz\" (UID: \"be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b\") " pod="openshift-ingress-canary/ingress-canary-md9qz"
Sep 17 16:42:06.271622 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.271550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.271622 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.271585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1bd0b-5754-49fd-84c1-28f2383539aa-config-volume\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.279559 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.279523    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:06.372331 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.372280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1bd0b-5754-49fd-84c1-28f2383539aa-config-volume\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.372331 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.372308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde1bd0b-5754-49fd-84c1-28f2383539aa-tmp-dir\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.372483 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.372334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmttr\" (UniqueName: \"kubernetes.io/projected/fde1bd0b-5754-49fd-84c1-28f2383539aa-kube-api-access-hmttr\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.372483 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.372357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b-cert\") pod \"ingress-canary-md9qz\" (UID: \"be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b\") " pod="openshift-ingress-canary/ingress-canary-md9qz"
Sep 17 16:42:06.372583 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.372532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwtbb\" (UniqueName: \"kubernetes.io/projected/be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b-kube-api-access-mwtbb\") pod \"ingress-canary-md9qz\" (UID: \"be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b\") " pod="openshift-ingress-canary/ingress-canary-md9qz"
Sep 17 16:42:06.372583 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.372566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.372674 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.372604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde1bd0b-5754-49fd-84c1-28f2383539aa-tmp-dir\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.372725 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:06.372705    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:06.372773 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:06.372764    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls podName:fde1bd0b-5754-49fd-84c1-28f2383539aa nodeName:}" failed. No retries permitted until 2026-09-17 16:42:06.872749906 +0000 UTC m=+35.584757153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls") pod "dns-default-9kh8p" (UID: "fde1bd0b-5754-49fd-84c1-28f2383539aa") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:06.372925 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.372902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1bd0b-5754-49fd-84c1-28f2383539aa-config-volume\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.376322 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.376303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b-cert\") pod \"ingress-canary-md9qz\" (UID: \"be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b\") " pod="openshift-ingress-canary/ingress-canary-md9qz"
Sep 17 16:42:06.380638 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.380619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwtbb\" (UniqueName: \"kubernetes.io/projected/be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b-kube-api-access-mwtbb\") pod \"ingress-canary-md9qz\" (UID: \"be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b\") " pod="openshift-ingress-canary/ingress-canary-md9qz"
Sep 17 16:42:06.380837 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.380820    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmttr\" (UniqueName: \"kubernetes.io/projected/fde1bd0b-5754-49fd-84c1-28f2383539aa-kube-api-access-hmttr\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.413950 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.413931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-md9qz"
Sep 17 16:42:06.698114 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.698047    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-md9qz"]
Sep 17 16:42:06.702487 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:06.702459    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9dad4e_fb00_4e83_bbf1_5cb7614d1e3b.slice/crio-15aa3b5df7badbae6768c2196d0933630bc33c14a695441b56a3cec3436d8534 WatchSource:0}: Error finding container 15aa3b5df7badbae6768c2196d0933630bc33c14a695441b56a3cec3436d8534: Status 404 returned error can't find the container with id 15aa3b5df7badbae6768c2196d0933630bc33c14a695441b56a3cec3436d8534
Sep 17 16:42:06.804369 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.804338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:42:06.804369 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.804351    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:42:06.804525 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.804351    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:42:06.807064 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.807043    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:42:06.808160 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.808103    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 16:42:06.808160 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.808109    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cc7hd\""
Sep 17 16:42:06.808160 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.808137    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:42:06.808160 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.808144    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lng7\""
Sep 17 16:42:06.808424 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.808112    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 16:42:06.877186 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:06.877163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:06.877291 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:06.877275    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:06.877346 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:06.877336    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls podName:fde1bd0b-5754-49fd-84c1-28f2383539aa nodeName:}" failed. No retries permitted until 2026-09-17 16:42:07.87731784 +0000 UTC m=+36.589325087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls") pod "dns-default-9kh8p" (UID: "fde1bd0b-5754-49fd-84c1-28f2383539aa") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:07.073017 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:07.072979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-md9qz" event={"ID":"be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b","Type":"ContainerStarted","Data":"15aa3b5df7badbae6768c2196d0933630bc33c14a695441b56a3cec3436d8534"}
Sep 17 16:42:07.886266 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:07.886211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:07.886691 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:07.886396    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:07.886691 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:07.886480    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls podName:fde1bd0b-5754-49fd-84c1-28f2383539aa nodeName:}" failed. No retries permitted until 2026-09-17 16:42:09.8864563 +0000 UTC m=+38.598463548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls") pod "dns-default-9kh8p" (UID: "fde1bd0b-5754-49fd-84c1-28f2383539aa") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:09.508406 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:09.508367    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:09.582437 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:09.582409    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:09.655667 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:09.655630    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:09.713457 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:09.713427    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:09.777671 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:09.777600    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:09.902926 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:09.902893    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:09.903012 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:09.903001    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:09.903059 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:09.903050    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls podName:fde1bd0b-5754-49fd-84c1-28f2383539aa nodeName:}" failed. No retries permitted until 2026-09-17 16:42:13.903037138 +0000 UTC m=+42.615044388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls") pod "dns-default-9kh8p" (UID: "fde1bd0b-5754-49fd-84c1-28f2383539aa") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:10.083106 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:10.083050    2566 generic.go:356] "Generic (PLEG): container finished" podID="48abb272-fae7-4755-a74e-f82be2c3b6f0" containerID="83e6d1981ad9b8ac3d1ff16f2e5c5f48ed555a6d6c20bf09243ade4e8ed0becb" exitCode=0
Sep 17 16:42:10.083207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:10.083125    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkvgc" event={"ID":"48abb272-fae7-4755-a74e-f82be2c3b6f0","Type":"ContainerDied","Data":"83e6d1981ad9b8ac3d1ff16f2e5c5f48ed555a6d6c20bf09243ade4e8ed0becb"}
Sep 17 16:42:10.084678 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:10.084658    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-md9qz" event={"ID":"be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b","Type":"ContainerStarted","Data":"28d2cc4e13b210c35fd6f07d7f56f6f94077bd391d5ae03a8c002fc03cf3f942"}
Sep 17 16:42:10.120103 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:10.120055    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-md9qz" podStartSLOduration=1.311960558 podStartE2EDuration="4.120041466s" totalImagesPullingTime="2.808080908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:10.120006941 +0000 UTC m=+38.832014210" watchObservedRunningTime="2026-09-17 16:42:10.120041466 +0000 UTC m=+38.832048733"
Sep 17 16:42:10.664902 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:10.664870    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:10.765171 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:10.765146    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:10.827970 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:10.827946    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:10.915904 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:10.915851    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:11.018001 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:11.017973    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:11.091556 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:11.091532    2566 generic.go:356] "Generic (PLEG): container finished" podID="48abb272-fae7-4755-a74e-f82be2c3b6f0" containerID="f08e33c80f3e31d382bedb57eeff7aeed28d78fe30976cf35622350a50f3940d" exitCode=0
Sep 17 16:42:11.091645 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:11.091613    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkvgc" event={"ID":"48abb272-fae7-4755-a74e-f82be2c3b6f0","Type":"ContainerDied","Data":"f08e33c80f3e31d382bedb57eeff7aeed28d78fe30976cf35622350a50f3940d"}
Sep 17 16:42:12.097449 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:12.097412    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkvgc" event={"ID":"48abb272-fae7-4755-a74e-f82be2c3b6f0","Type":"ContainerStarted","Data":"d1913e16b45a6a2989add9cbb7cc319d448fabb616ceeec1ef84b5c97f5530e6"}
Sep 17 16:42:12.119238 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:12.119195    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lkvgc" podStartSLOduration=12.7884466 podStartE2EDuration="40.119181s" totalImagesPullingTime="27.3307344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:12.117780476 +0000 UTC m=+40.829787744" watchObservedRunningTime="2026-09-17 16:42:12.119181 +0000 UTC m=+40.831188267"
Sep 17 16:42:12.244139 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:12.244108    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:12.345370 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:12.345329    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:13.931231 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:13.931198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:13.931646 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:13.931322    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:13.931646 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:13.931371    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls podName:fde1bd0b-5754-49fd-84c1-28f2383539aa nodeName:}" failed. No retries permitted until 2026-09-17 16:42:21.931358529 +0000 UTC m=+50.643365774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls") pod "dns-default-9kh8p" (UID: "fde1bd0b-5754-49fd-84c1-28f2383539aa") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:17.154643 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.154607    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"]
Sep 17 16:42:17.186339 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.186295    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"]
Sep 17 16:42:17.186491 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.186415    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"
Sep 17 16:42:17.189292 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.189267    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 16:42:17.189400 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.189302    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 16:42:17.189400 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.189318    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 16:42:17.190544 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.190527    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:42:17.190657 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.190552    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6mb78\""
Sep 17 16:42:17.244977 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.244957    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"]
Sep 17 16:42:17.266193 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.266173    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"]
Sep 17 16:42:17.266308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.266268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"
Sep 17 16:42:17.268937 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.268921    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 16:42:17.268937 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.268947    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 16:42:17.269126 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.269106    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 16:42:17.269619 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.269594    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:42:17.269731 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.269693    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tzr2b\""
Sep 17 16:42:17.353341 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.353319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3efa2ff4-6070-46a8-941d-0c42ee9f24e1-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-bl8n9\" (UID: \"3efa2ff4-6070-46a8-941d-0c42ee9f24e1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"
Sep 17 16:42:17.353459 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.353366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3efa2ff4-6070-46a8-941d-0c42ee9f24e1-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-bl8n9\" (UID: \"3efa2ff4-6070-46a8-941d-0c42ee9f24e1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"
Sep 17 16:42:17.353535 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.353469    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz86g\" (UniqueName: \"kubernetes.io/projected/3efa2ff4-6070-46a8-941d-0c42ee9f24e1-kube-api-access-jz86g\") pod \"kube-storage-version-migrator-operator-55889f7b47-bl8n9\" (UID: \"3efa2ff4-6070-46a8-941d-0c42ee9f24e1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"
Sep 17 16:42:17.354700 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.354678    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b7d668668-7nwdq"]
Sep 17 16:42:17.378703 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.378683    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"]
Sep 17 16:42:17.378825 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.378810    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.381685 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.381664    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 16:42:17.381685 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.381682    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ljc5n\""
Sep 17 16:42:17.381842 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.381729    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 16:42:17.381842 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.381729    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 16:42:17.381842 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.381834    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 16:42:17.382110 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.381890    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 16:42:17.382169 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.382156    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 16:42:17.396664 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.396643    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"]
Sep 17 16:42:17.396742 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.396670    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b7d668668-7nwdq"]
Sep 17 16:42:17.396779 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.396753    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:17.399348 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.399331    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 16:42:17.399614 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.399598    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-br2rp\""
Sep 17 16:42:17.399696 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.399628    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 16:42:17.399696 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.399604    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 16:42:17.403470 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.403442    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 16:42:17.447146 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.447100    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7ffb56c84c-bqjhr"]
Sep 17 16:42:17.454748 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.454730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ada1cced-1b84-44be-bcf1-4dc82bed86ec-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:17.454854 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.454774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3efa2ff4-6070-46a8-941d-0c42ee9f24e1-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-bl8n9\" (UID: \"3efa2ff4-6070-46a8-941d-0c42ee9f24e1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"
Sep 17 16:42:17.454916 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.454858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80a022e2-e6f2-4047-9d8c-83b38fdece2a-serving-cert\") pod \"service-ca-operator-57cdc4b768-zgv27\" (UID: \"80a022e2-e6f2-4047-9d8c-83b38fdece2a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"
Sep 17 16:42:17.454916 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.454884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:17.454916 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.454904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2l7b\" (UniqueName: \"kubernetes.io/projected/80a022e2-e6f2-4047-9d8c-83b38fdece2a-kube-api-access-x2l7b\") pod \"service-ca-operator-57cdc4b768-zgv27\" (UID: \"80a022e2-e6f2-4047-9d8c-83b38fdece2a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"
Sep 17 16:42:17.455068 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.454921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.455068 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.454979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz86g\" (UniqueName: \"kubernetes.io/projected/3efa2ff4-6070-46a8-941d-0c42ee9f24e1-kube-api-access-jz86g\") pod \"kube-storage-version-migrator-operator-55889f7b47-bl8n9\" (UID: \"3efa2ff4-6070-46a8-941d-0c42ee9f24e1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"
Sep 17 16:42:17.455068 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.455010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80a022e2-e6f2-4047-9d8c-83b38fdece2a-config\") pod \"service-ca-operator-57cdc4b768-zgv27\" (UID: \"80a022e2-e6f2-4047-9d8c-83b38fdece2a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"
Sep 17 16:42:17.455068 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.455028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-stats-auth\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.455068 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.455052    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.455327 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.455071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzsm\" (UniqueName: \"kubernetes.io/projected/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-kube-api-access-9tzsm\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.455327 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.455091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktlgf\" (UniqueName: \"kubernetes.io/projected/ada1cced-1b84-44be-bcf1-4dc82bed86ec-kube-api-access-ktlgf\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:17.455327 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.455145    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3efa2ff4-6070-46a8-941d-0c42ee9f24e1-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-bl8n9\" (UID: \"3efa2ff4-6070-46a8-941d-0c42ee9f24e1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"
Sep 17 16:42:17.455327 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.455170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-default-certificate\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.455613 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.455598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3efa2ff4-6070-46a8-941d-0c42ee9f24e1-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-bl8n9\" (UID: \"3efa2ff4-6070-46a8-941d-0c42ee9f24e1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"
Sep 17 16:42:17.457715 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.457695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3efa2ff4-6070-46a8-941d-0c42ee9f24e1-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-bl8n9\" (UID: \"3efa2ff4-6070-46a8-941d-0c42ee9f24e1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"
Sep 17 16:42:17.466441 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.466424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz86g\" (UniqueName: \"kubernetes.io/projected/3efa2ff4-6070-46a8-941d-0c42ee9f24e1-kube-api-access-jz86g\") pod \"kube-storage-version-migrator-operator-55889f7b47-bl8n9\" (UID: \"3efa2ff4-6070-46a8-941d-0c42ee9f24e1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"
Sep 17 16:42:17.469952 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.469932    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7ffb56c84c-bqjhr"]
Sep 17 16:42:17.470022 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.470011    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.472943 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.472922    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nw7ln\""
Sep 17 16:42:17.473012 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.472930    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 16:42:17.473200 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.473184    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 16:42:17.473376 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.473342    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 16:42:17.480865 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.480845    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 16:42:17.495413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.495370    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"
Sep 17 16:42:17.522014 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.521989    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:17.556099 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2l7b\" (UniqueName: \"kubernetes.io/projected/80a022e2-e6f2-4047-9d8c-83b38fdece2a-kube-api-access-x2l7b\") pod \"service-ca-operator-57cdc4b768-zgv27\" (UID: \"80a022e2-e6f2-4047-9d8c-83b38fdece2a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"
Sep 17 16:42:17.556324 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-image-registry-private-configuration\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.556324 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556275    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kz9k\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-kube-api-access-7kz9k\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.556324 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556312    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.556495 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-bound-sa-token\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.556495 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80a022e2-e6f2-4047-9d8c-83b38fdece2a-config\") pod \"service-ca-operator-57cdc4b768-zgv27\" (UID: \"80a022e2-e6f2-4047-9d8c-83b38fdece2a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"
Sep 17 16:42:17.556495 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-stats-auth\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.556635 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.556635 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzsm\" (UniqueName: \"kubernetes.io/projected/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-kube-api-access-9tzsm\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.556635 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556562    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-ca-trust-extracted\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.556635 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktlgf\" (UniqueName: \"kubernetes.io/projected/ada1cced-1b84-44be-bcf1-4dc82bed86ec-kube-api-access-ktlgf\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:17.556635 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-trusted-ca\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.556881 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.556881 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-certificates\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.556881 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-installation-pull-secrets\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.556881 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-default-certificate\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.556881 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ada1cced-1b84-44be-bcf1-4dc82bed86ec-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:17.557128 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80a022e2-e6f2-4047-9d8c-83b38fdece2a-serving-cert\") pod \"service-ca-operator-57cdc4b768-zgv27\" (UID: \"80a022e2-e6f2-4047-9d8c-83b38fdece2a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"
Sep 17 16:42:17.557128 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.556915    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:17.557128 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:17.557018    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:17.557128 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:17.557062    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls podName:ada1cced-1b84-44be-bcf1-4dc82bed86ec nodeName:}" failed. No retries permitted until 2026-09-17 16:42:18.057044943 +0000 UTC m=+46.769052189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qd8v9" (UID: "ada1cced-1b84-44be-bcf1-4dc82bed86ec") : secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:17.557128 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:17.557080    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:18.057073612 +0000 UTC m=+46.769080858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : configmap references non-existent config key: service-ca.crt
Sep 17 16:42:17.557542 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.557515    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80a022e2-e6f2-4047-9d8c-83b38fdece2a-config\") pod \"service-ca-operator-57cdc4b768-zgv27\" (UID: \"80a022e2-e6f2-4047-9d8c-83b38fdece2a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"
Sep 17 16:42:17.557664 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:17.557644    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 16:42:17.557766 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:17.557724    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:18.057702325 +0000 UTC m=+46.769709572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : secret "router-metrics-certs-default" not found
Sep 17 16:42:17.559071 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.558421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ada1cced-1b84-44be-bcf1-4dc82bed86ec-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:17.560545 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.560520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-stats-auth\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.561206 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.561100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-default-certificate\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.561626 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.561601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80a022e2-e6f2-4047-9d8c-83b38fdece2a-serving-cert\") pod \"service-ca-operator-57cdc4b768-zgv27\" (UID: \"80a022e2-e6f2-4047-9d8c-83b38fdece2a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"
Sep 17 16:42:17.577458 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.577439    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2l7b\" (UniqueName: \"kubernetes.io/projected/80a022e2-e6f2-4047-9d8c-83b38fdece2a-kube-api-access-x2l7b\") pod \"service-ca-operator-57cdc4b768-zgv27\" (UID: \"80a022e2-e6f2-4047-9d8c-83b38fdece2a\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"
Sep 17 16:42:17.577545 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.577514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktlgf\" (UniqueName: \"kubernetes.io/projected/ada1cced-1b84-44be-bcf1-4dc82bed86ec-kube-api-access-ktlgf\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:17.578080 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.578056    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzsm\" (UniqueName: \"kubernetes.io/projected/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-kube-api-access-9tzsm\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:17.622298 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.622244    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:17.657584 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.657562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-image-registry-private-configuration\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.657665 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.657591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kz9k\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-kube-api-access-7kz9k\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.657665 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.657615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-bound-sa-token\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.657825 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.657807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-ca-trust-extracted\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.657861 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.657843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-trusted-ca\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.657901 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.657862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.657901 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.657879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-certificates\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.657901 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.657895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-installation-pull-secrets\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.658041 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:17.657961    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:17.658041 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:17.657982    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ffb56c84c-bqjhr: secret "image-registry-tls" not found
Sep 17 16:42:17.658128 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:17.658054    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls podName:dd1f5aae-b2a9-4f09-8fef-0572bf038c72 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:18.158035168 +0000 UTC m=+46.870042433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls") pod "image-registry-7ffb56c84c-bqjhr" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72") : secret "image-registry-tls" not found
Sep 17 16:42:17.658194 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.658157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-ca-trust-extracted\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.658524 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.658500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-certificates\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.659031 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.659007    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-trusted-ca\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.659938 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.659920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-installation-pull-secrets\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.660073 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.660057    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-image-registry-private-configuration\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.668383 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.668365    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kz9k\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-kube-api-access-7kz9k\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.668613 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.668597    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-bound-sa-token\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:17.777673 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.777650    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9"]
Sep 17 16:42:17.780443 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:17.780425    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efa2ff4_6070_46a8_941d_0c42ee9f24e1.slice/crio-055c6cd5719c05459b4b65d12f6944a185536df6ef05a732985813c58208667c WatchSource:0}: Error finding container 055c6cd5719c05459b4b65d12f6944a185536df6ef05a732985813c58208667c: Status 404 returned error can't find the container with id 055c6cd5719c05459b4b65d12f6944a185536df6ef05a732985813c58208667c
Sep 17 16:42:17.874360 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:17.874338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"
Sep 17 16:42:18.060926 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:18.060849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:18.060926 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:18.060894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:18.061143 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:18.060953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:18.061143 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:18.061022    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:18.061143 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:18.061100    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls podName:ada1cced-1b84-44be-bcf1-4dc82bed86ec nodeName:}" failed. No retries permitted until 2026-09-17 16:42:19.061080348 +0000 UTC m=+47.773087611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qd8v9" (UID: "ada1cced-1b84-44be-bcf1-4dc82bed86ec") : secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:18.061143 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:18.061122    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:19.061111351 +0000 UTC m=+47.773118600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : configmap references non-existent config key: service-ca.crt
Sep 17 16:42:18.061143 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:18.061100    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 16:42:18.061421 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:18.061196    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:19.061178567 +0000 UTC m=+47.773185818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : secret "router-metrics-certs-default" not found
Sep 17 16:42:18.112232 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:18.112204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9" event={"ID":"3efa2ff4-6070-46a8-941d-0c42ee9f24e1","Type":"ContainerStarted","Data":"055c6cd5719c05459b4b65d12f6944a185536df6ef05a732985813c58208667c"}
Sep 17 16:42:18.161809 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:18.161785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:18.162088 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:18.161923    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:18.162088 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:18.161938    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ffb56c84c-bqjhr: secret "image-registry-tls" not found
Sep 17 16:42:18.162088 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:18.161980    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls podName:dd1f5aae-b2a9-4f09-8fef-0572bf038c72 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:19.161966936 +0000 UTC m=+47.873974187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls") pod "image-registry-7ffb56c84c-bqjhr" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72") : secret "image-registry-tls" not found
Sep 17 16:42:18.168412 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:18.168392    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27"]
Sep 17 16:42:18.170970 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:18.170949    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a022e2_e6f2_4047_9d8c_83b38fdece2a.slice/crio-4055d2ef2b2d7ad6cca43743912bcfc56ec7beb5873e72f74b52a0e43cccfd91 WatchSource:0}: Error finding container 4055d2ef2b2d7ad6cca43743912bcfc56ec7beb5873e72f74b52a0e43cccfd91: Status 404 returned error can't find the container with id 4055d2ef2b2d7ad6cca43743912bcfc56ec7beb5873e72f74b52a0e43cccfd91
Sep 17 16:42:19.069422 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:19.069385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:19.069594 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:19.069466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:19.069594 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:19.069527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:19.069594 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:19.069562    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:19.069766 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:19.069613    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 16:42:19.069766 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:19.069629    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls podName:ada1cced-1b84-44be-bcf1-4dc82bed86ec nodeName:}" failed. No retries permitted until 2026-09-17 16:42:21.069608392 +0000 UTC m=+49.781615640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qd8v9" (UID: "ada1cced-1b84-44be-bcf1-4dc82bed86ec") : secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:19.069766 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:19.069648    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:21.069638699 +0000 UTC m=+49.781645951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : configmap references non-existent config key: service-ca.crt
Sep 17 16:42:19.069766 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:19.069708    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:21.069698127 +0000 UTC m=+49.781705378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : secret "router-metrics-certs-default" not found
Sep 17 16:42:19.117208 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:19.117145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27" event={"ID":"80a022e2-e6f2-4047-9d8c-83b38fdece2a","Type":"ContainerStarted","Data":"4055d2ef2b2d7ad6cca43743912bcfc56ec7beb5873e72f74b52a0e43cccfd91"}
Sep 17 16:42:19.170207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:19.170170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:19.170659 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:19.170348    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:19.170659 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:19.170365    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ffb56c84c-bqjhr: secret "image-registry-tls" not found
Sep 17 16:42:19.170659 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:19.170427    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls podName:dd1f5aae-b2a9-4f09-8fef-0572bf038c72 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:21.170407928 +0000 UTC m=+49.882415200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls") pod "image-registry-7ffb56c84c-bqjhr" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72") : secret "image-registry-tls" not found
Sep 17 16:42:19.768391 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:19.768355    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:19.831707 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:19.831668    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:21.086686 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.086663    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:21.086937 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.086764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:21.086937 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.086798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:21.086937 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:21.086820    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 16:42:21.086937 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:21.086881    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:25.086861973 +0000 UTC m=+53.798869218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : secret "router-metrics-certs-default" not found
Sep 17 16:42:21.086937 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:21.086887    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:21.086937 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:21.086909    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:25.086899905 +0000 UTC m=+53.798907158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : configmap references non-existent config key: service-ca.crt
Sep 17 16:42:21.087124 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:21.086958    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls podName:ada1cced-1b84-44be-bcf1-4dc82bed86ec nodeName:}" failed. No retries permitted until 2026-09-17 16:42:25.08694388 +0000 UTC m=+53.798951127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qd8v9" (UID: "ada1cced-1b84-44be-bcf1-4dc82bed86ec") : secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:21.187327 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.187305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:21.187462 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:21.187441    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:21.187614 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:21.187463    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ffb56c84c-bqjhr: secret "image-registry-tls" not found
Sep 17 16:42:21.187614 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:21.187522    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls podName:dd1f5aae-b2a9-4f09-8fef-0572bf038c72 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:25.187500456 +0000 UTC m=+53.899507709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls") pod "image-registry-7ffb56c84c-bqjhr" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72") : secret "image-registry-tls" not found
Sep 17 16:42:21.407226 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.407129    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:21.473717 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.473676    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:21.865808 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.865769    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:21.872736 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.872712    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qzw8z"]
Sep 17 16:42:21.897493 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.897473    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qzw8z"]
Sep 17 16:42:21.897609 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.897597    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:21.900336 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.900316    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 16:42:21.900429 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.900316    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-grs8h\""
Sep 17 16:42:21.900772 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.900755    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 16:42:21.901179 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.901162    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 16:42:21.901287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.901166    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 16:42:21.977152 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.977095    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:21.994959 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.994929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:21.995080 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.994979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd5495c6-27d1-4d2f-a513-52453bc60ca2-data-volume\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:21.995145 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.995110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:21.995197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.995146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd5495c6-27d1-4d2f-a513-52453bc60ca2-crio-socket\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:21.995197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.995180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:21.995318 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:21.995223    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:21.995318 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:21.995223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gdnk\" (UniqueName: \"kubernetes.io/projected/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-api-access-4gdnk\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:21.995318 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:21.995292    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls podName:fde1bd0b-5754-49fd-84c1-28f2383539aa nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.995274656 +0000 UTC m=+66.707281906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls") pod "dns-default-9kh8p" (UID: "fde1bd0b-5754-49fd-84c1-28f2383539aa") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:22.095801 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.095772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:22.096169 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.095827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd5495c6-27d1-4d2f-a513-52453bc60ca2-data-volume\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:22.096169 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:22.095882    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:22.096169 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:22.095932    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm podName:dd5495c6-27d1-4d2f-a513-52453bc60ca2 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:22.595916035 +0000 UTC m=+51.307923301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qzw8z" (UID: "dd5495c6-27d1-4d2f-a513-52453bc60ca2") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:22.096169 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.095951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd5495c6-27d1-4d2f-a513-52453bc60ca2-crio-socket\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:22.096169 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.095988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:22.096169 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.096025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gdnk\" (UniqueName: \"kubernetes.io/projected/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-api-access-4gdnk\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:22.096169 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:22.096100    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:22.096169 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.096153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd5495c6-27d1-4d2f-a513-52453bc60ca2-crio-socket\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:22.096169 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:22.096169    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls podName:dd5495c6-27d1-4d2f-a513-52453bc60ca2 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:22.596154354 +0000 UTC m=+51.308161606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzw8z" (UID: "dd5495c6-27d1-4d2f-a513-52453bc60ca2") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:22.096640 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.096200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd5495c6-27d1-4d2f-a513-52453bc60ca2-data-volume\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:22.104579 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.104548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gdnk\" (UniqueName: \"kubernetes.io/projected/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-api-access-4gdnk\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:22.126336 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.126269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27" event={"ID":"80a022e2-e6f2-4047-9d8c-83b38fdece2a","Type":"ContainerStarted","Data":"ef866c52b2877017c6cbc049e4259e55d0d9bceb88ebb5a9be0dc359fe3ff109"}
Sep 17 16:42:22.127907 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.127884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9" event={"ID":"3efa2ff4-6070-46a8-941d-0c42ee9f24e1","Type":"ContainerStarted","Data":"7130158ca670ccd99b8cb6fee34913465aa0717e1e931114c347c0b2ba79153a"}
Sep 17 16:42:22.146728 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.144154    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27" podStartSLOduration=2.198438015 podStartE2EDuration="5.14413637s" totalImagesPullingTime="2.945698355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:22.14150151 +0000 UTC m=+50.853508778" watchObservedRunningTime="2026-09-17 16:42:22.14413637 +0000 UTC m=+50.856143639"
Sep 17 16:42:22.160468 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.160430    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9" podStartSLOduration=1.831183132 podStartE2EDuration="5.160416181s" totalImagesPullingTime="3.329233049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:22.159431652 +0000 UTC m=+50.871438920" watchObservedRunningTime="2026-09-17 16:42:22.160416181 +0000 UTC m=+50.872423450"
Sep 17 16:42:22.523983 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.523952    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:22.591772 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.591740    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:22.599717 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.599682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:22.599832 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.599739    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:22.599832 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:22.599811    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:22.599832 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:22.599812    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:22.599996 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:22.599863    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls podName:dd5495c6-27d1-4d2f-a513-52453bc60ca2 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:23.599849776 +0000 UTC m=+52.311857027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzw8z" (UID: "dd5495c6-27d1-4d2f-a513-52453bc60ca2") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:22.599996 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:22.599877    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm podName:dd5495c6-27d1-4d2f-a513-52453bc60ca2 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:23.599870309 +0000 UTC m=+52.311877555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qzw8z" (UID: "dd5495c6-27d1-4d2f-a513-52453bc60ca2") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:22.706346 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.706317    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:22.781946 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.781867    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:22.881371 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.881345    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:22.936476 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:22.936445    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:23.526149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:23.526120    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:23.608213 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:23.608188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:23.608369 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:23.608242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:23.608369 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:23.608315    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:23.608369 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:23.608345    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:23.608369 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:23.608365    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm podName:dd5495c6-27d1-4d2f-a513-52453bc60ca2 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:25.608351867 +0000 UTC m=+54.320359113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qzw8z" (UID: "dd5495c6-27d1-4d2f-a513-52453bc60ca2") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:23.608509 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:23.608404    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls podName:dd5495c6-27d1-4d2f-a513-52453bc60ca2 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:25.608389958 +0000 UTC m=+54.320397203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzw8z" (UID: "dd5495c6-27d1-4d2f-a513-52453bc60ca2") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:23.629332 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:23.629302    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:23.797775 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:23.797723    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8hvh7_55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe/dns-node-resolver/0.log"
Sep 17 16:42:25.000239 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:25.000211    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h774k_89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10/node-ca/0.log"
Sep 17 16:42:25.119837 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:25.119808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:25.119962 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:25.119848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:25.119962 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:25.119892    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:25.120038 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.119974    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:33.119954337 +0000 UTC m=+61.831961584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : configmap references non-existent config key: service-ca.crt
Sep 17 16:42:25.120038 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.119980    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:25.120038 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.120015    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls podName:ada1cced-1b84-44be-bcf1-4dc82bed86ec nodeName:}" failed. No retries permitted until 2026-09-17 16:42:33.120007197 +0000 UTC m=+61.832014442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qd8v9" (UID: "ada1cced-1b84-44be-bcf1-4dc82bed86ec") : secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:25.120038 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.119980    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 16:42:25.120038 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.120037    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:33.120031723 +0000 UTC m=+61.832038969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : secret "router-metrics-certs-default" not found
Sep 17 16:42:25.180538 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:25.180515    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:25.220734 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:25.220710    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:25.220863 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.220840    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:25.220863 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.220857    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ffb56c84c-bqjhr: secret "image-registry-tls" not found
Sep 17 16:42:25.220972 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.220914    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls podName:dd1f5aae-b2a9-4f09-8fef-0572bf038c72 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:33.220895395 +0000 UTC m=+61.932902661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls") pod "image-registry-7ffb56c84c-bqjhr" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72") : secret "image-registry-tls" not found
Sep 17 16:42:25.282413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:25.282355    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:25.598485 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:25.598417    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-md9qz_be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b/serve-healthcheck-canary/0.log"
Sep 17 16:42:25.623843 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:25.623814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:25.623949 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:25.623869    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:25.623949 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.623940    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:25.624022 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.623954    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:25.624022 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.623997    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm podName:dd5495c6-27d1-4d2f-a513-52453bc60ca2 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:29.623982214 +0000 UTC m=+58.335989483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qzw8z" (UID: "dd5495c6-27d1-4d2f-a513-52453bc60ca2") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:25.624022 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:25.624012    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls podName:dd5495c6-27d1-4d2f-a513-52453bc60ca2 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:29.624004733 +0000 UTC m=+58.336011979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzw8z" (UID: "dd5495c6-27d1-4d2f-a513-52453bc60ca2") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:26.296311 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:26.296277    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:27.449459 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:27.449419    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:27.552737 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:27.552712    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:29.652536 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:29.652492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:29.652997 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:29.652614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:29.652997 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:29.652624    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:29.652997 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:29.652693    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm podName:dd5495c6-27d1-4d2f-a513-52453bc60ca2 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.652676551 +0000 UTC m=+66.364683797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qzw8z" (UID: "dd5495c6-27d1-4d2f-a513-52453bc60ca2") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:29.652997 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:29.652727    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:29.652997 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:29.652774    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls podName:dd5495c6-27d1-4d2f-a513-52453bc60ca2 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.652761398 +0000 UTC m=+66.364768648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzw8z" (UID: "dd5495c6-27d1-4d2f-a513-52453bc60ca2") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:29.855433 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:29.855399    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:30.068062 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:30.068037    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:30.279103 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:30.279067    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:30.379875 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:30.379809    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:30.439832 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:30.439805    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:30.622248 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:30.622218    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:30.689178 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:30.689110    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:30.769226 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:30.769197    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:31.075655 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:31.075622    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxtcx"
Sep 17 16:42:31.359281 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:31.359200    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:31.956125 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:31.956097    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:32.156053 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:32.156019    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:32.362934 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:32.362894    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:32.762575 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:32.762540    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:33.178944 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:33.178859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:33.178944 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:33.178906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:33.179399 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:33.178966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:33.179399 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:33.179009    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:49.178990192 +0000 UTC m=+77.890997437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : configmap references non-existent config key: service-ca.crt
Sep 17 16:42:33.179399 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:33.179069    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:33.179399 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:33.179099    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 16:42:33.179399 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:33.179119    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls podName:ada1cced-1b84-44be-bcf1-4dc82bed86ec nodeName:}" failed. No retries permitted until 2026-09-17 16:42:49.17910631 +0000 UTC m=+77.891113555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qd8v9" (UID: "ada1cced-1b84-44be-bcf1-4dc82bed86ec") : secret "cluster-monitoring-operator-tls" not found
Sep 17 16:42:33.179399 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:33.179142    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs podName:9b164a3b-fc9f-48c7-a6ac-c2e3012379b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:49.179129136 +0000 UTC m=+77.891136382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs") pod "router-default-b7d668668-7nwdq" (UID: "9b164a3b-fc9f-48c7-a6ac-c2e3012379b8") : secret "router-metrics-certs-default" not found
Sep 17 16:42:33.279855 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:33.279830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:33.279982 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:33.279959    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:33.279982 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:33.279970    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ffb56c84c-bqjhr: secret "image-registry-tls" not found
Sep 17 16:42:33.280053 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:33.280009    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls podName:dd1f5aae-b2a9-4f09-8fef-0572bf038c72 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:49.279995784 +0000 UTC m=+77.992003038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls") pod "image-registry-7ffb56c84c-bqjhr" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72") : secret "image-registry-tls" not found
Sep 17 16:42:33.451586 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:33.451512    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:33.575577 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:33.575538    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:33.675268 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:33.675225    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:33.968081 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:33.968041    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:34.156801 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:34.156770    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:34.398714 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:34.398685    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:34.464181 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:34.464153    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:34.763594 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:34.763560    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:35.016194 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:35.016128    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:35.371566 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:35.371488    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:35.761081 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:35.761038    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:36.123390 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:36.123314    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:36.187853 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:36.187825    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:36.560677 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:36.560648    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:36.779951 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:36.779919    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:36.958435 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:36.958370    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:37.511449 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.511346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:42:37.511449 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.511434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:42:37.514346 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.514324    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:42:37.514346 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.514337    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:42:37.522373 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:37.522338    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 16:42:37.522527 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:37.522403    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs podName:4efac150-5db6-42c1-8309-138120e7f820 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:41.522386184 +0000 UTC m=+130.234393436 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs") pod "network-metrics-daemon-vm27c" (UID: "4efac150-5db6-42c1-8309-138120e7f820") : secret "metrics-daemon-secret" not found
Sep 17 16:42:37.524716 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.524694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df7d88b0-b648-45ed-8381-e7172a404315-original-pull-secret\") pod \"global-pull-secret-syncer-qw97h\" (UID: \"df7d88b0-b648-45ed-8381-e7172a404315\") " pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:42:37.612235 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.612212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft45g\" (UniqueName: \"kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g\") pod \"network-check-target-w6lnl\" (UID: \"8a1950af-8604-4048-93b4-1bdb996f291e\") " pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:42:37.615442 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.615426    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 16:42:37.625717 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.625693    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 16:42:37.635246 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.635216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft45g\" (UniqueName: \"kubernetes.io/projected/8a1950af-8604-4048-93b4-1bdb996f291e-kube-api-access-ft45g\") pod \"network-check-target-w6lnl\" (UID: \"8a1950af-8604-4048-93b4-1bdb996f291e\") " pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:42:37.713452 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.713428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:37.713555 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.713542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:37.713683 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:37.713666    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:37.713730 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:37.713721    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls podName:dd5495c6-27d1-4d2f-a513-52453bc60ca2 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:53.713706275 +0000 UTC m=+82.425713521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qzw8z" (UID: "dd5495c6-27d1-4d2f-a513-52453bc60ca2") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:37.713866 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.713849    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd5495c6-27d1-4d2f-a513-52453bc60ca2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:37.724881 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.724864    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qw97h"
Sep 17 16:42:37.732192 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.732101    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cc7hd\""
Sep 17 16:42:37.740384 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.740368    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:42:37.774932 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.774831    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:37.996754 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:37.996724    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qw97h"]
Sep 17 16:42:37.999913 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:37.999885    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7d88b0_b648_45ed_8381_e7172a404315.slice/crio-12bba848268cb9538bcf7f7af91ed15d41dd9c6fffdfb3d4d122c53fc87f5cb9 WatchSource:0}: Error finding container 12bba848268cb9538bcf7f7af91ed15d41dd9c6fffdfb3d4d122c53fc87f5cb9: Status 404 returned error can't find the container with id 12bba848268cb9538bcf7f7af91ed15d41dd9c6fffdfb3d4d122c53fc87f5cb9
Sep 17 16:42:38.015097 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.015066    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w6lnl"]
Sep 17 16:42:38.015925 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.015904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:38.018164 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.018135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde1bd0b-5754-49fd-84c1-28f2383539aa-metrics-tls\") pod \"dns-default-9kh8p\" (UID: \"fde1bd0b-5754-49fd-84c1-28f2383539aa\") " pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:38.020465 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:38.020441    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1950af_8604_4048_93b4_1bdb996f291e.slice/crio-04e6f72338531bead65a46314502659178240d6ffce8a9b26187115bd709960e WatchSource:0}: Error finding container 04e6f72338531bead65a46314502659178240d6ffce8a9b26187115bd709960e: Status 404 returned error can't find the container with id 04e6f72338531bead65a46314502659178240d6ffce8a9b26187115bd709960e
Sep 17 16:42:38.165299 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.165269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qw97h" event={"ID":"df7d88b0-b648-45ed-8381-e7172a404315","Type":"ContainerStarted","Data":"12bba848268cb9538bcf7f7af91ed15d41dd9c6fffdfb3d4d122c53fc87f5cb9"}
Sep 17 16:42:38.166501 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.166481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w6lnl" event={"ID":"8a1950af-8604-4048-93b4-1bdb996f291e","Type":"ContainerStarted","Data":"04e6f72338531bead65a46314502659178240d6ffce8a9b26187115bd709960e"}
Sep 17 16:42:38.210596 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.210576    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xc8sp\""
Sep 17 16:42:38.218628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.218608    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:38.365927 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.365861    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:38.488728 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.488695    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9kh8p"]
Sep 17 16:42:38.492079 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:38.492052    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde1bd0b_5754_49fd_84c1_28f2383539aa.slice/crio-4422affb3c1a1c3e48b6e4aec2c77227f86976ec70f741f7e77cd0243ce5db49 WatchSource:0}: Error finding container 4422affb3c1a1c3e48b6e4aec2c77227f86976ec70f741f7e77cd0243ce5db49: Status 404 returned error can't find the container with id 4422affb3c1a1c3e48b6e4aec2c77227f86976ec70f741f7e77cd0243ce5db49
Sep 17 16:42:38.563796 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.563753    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:38.714048 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.713927    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:38.777448 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.777410    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:38.966794 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:38.966715    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:39.172025 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:39.171989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9kh8p" event={"ID":"fde1bd0b-5754-49fd-84c1-28f2383539aa","Type":"ContainerStarted","Data":"4422affb3c1a1c3e48b6e4aec2c77227f86976ec70f741f7e77cd0243ce5db49"}
Sep 17 16:42:39.963459 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:39.963348    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:40.159796 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:40.159761    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:40.367308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:40.367203    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:40.558642 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:40.558557    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:40.936598 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:40.936519    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:41.005061 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:41.005024    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:41.181649 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:41.181612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9kh8p" event={"ID":"fde1bd0b-5754-49fd-84c1-28f2383539aa","Type":"ContainerStarted","Data":"fcf6ee9b6db83a9de8c0605c851e0ce1583b9ca570d89ba23e0a9223b960822e"}
Sep 17 16:42:41.181834 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:41.181681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9kh8p" event={"ID":"fde1bd0b-5754-49fd-84c1-28f2383539aa","Type":"ContainerStarted","Data":"2e744388ed58aded029fcb840d30a3a99bc350d3247796425b7e66bdf6f1e28d"}
Sep 17 16:42:41.181834 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:41.181715    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:41.200029 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:41.199937    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9kh8p" podStartSLOduration=33.429092494 podStartE2EDuration="35.199918877s" totalImagesPullingTime="1.770826383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:41.198365215 +0000 UTC m=+69.910372494" watchObservedRunningTime="2026-09-17 16:42:41.199918877 +0000 UTC m=+69.911926151"
Sep 17 16:42:44.193182 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:44.193138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w6lnl" event={"ID":"8a1950af-8604-4048-93b4-1bdb996f291e","Type":"ContainerStarted","Data":"53af1d19a7069b790001c68dbbb594cfabdf6237c42cab36d50f1d8c8d163c48"}
Sep 17 16:42:44.193667 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:44.193267    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:42:44.194916 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:44.194888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qw97h" event={"ID":"df7d88b0-b648-45ed-8381-e7172a404315","Type":"ContainerStarted","Data":"efe359be0467f00b7f693e3e178df5cf0bfd3218c26e80762c64a0d43b3227a8"}
Sep 17 16:42:44.208985 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:44.208939    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w6lnl" podStartSLOduration=67.024046639 podStartE2EDuration="1m12.208927129s" totalImagesPullingTime="5.18488049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:44.208729724 +0000 UTC m=+72.920736995" watchObservedRunningTime="2026-09-17 16:42:44.208927129 +0000 UTC m=+72.920934398"
Sep 17 16:42:44.224677 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:44.224640    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qw97h" podStartSLOduration=67.009413771 podStartE2EDuration="1m12.224628282s" totalImagesPullingTime="5.215214511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:44.22450767 +0000 UTC m=+72.936514939" watchObservedRunningTime="2026-09-17 16:42:44.224628282 +0000 UTC m=+72.936635549"
Sep 17 16:42:49.203386 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.203344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:49.203386 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.203395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:49.204011 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.203461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:49.204011 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.203967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-service-ca-bundle\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:49.205901 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.205869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b164a3b-fc9f-48c7-a6ac-c2e3012379b8-metrics-certs\") pod \"router-default-b7d668668-7nwdq\" (UID: \"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8\") " pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:49.205901 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.205869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ada1cced-1b84-44be-bcf1-4dc82bed86ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qd8v9\" (UID: \"ada1cced-1b84-44be-bcf1-4dc82bed86ec\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:49.304391 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.304358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:49.306602 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.306583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls\") pod \"image-registry-7ffb56c84c-bqjhr\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") " pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:49.489115 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.489085    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ljc5n\""
Sep 17 16:42:49.496822 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.496798    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:49.508546 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.508523    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-br2rp\""
Sep 17 16:42:49.517422 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.517400    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"
Sep 17 16:42:49.582807 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.582595    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nw7ln\""
Sep 17 16:42:49.594560 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.594239    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:49.768757 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.768698    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b7d668668-7nwdq"]
Sep 17 16:42:49.771755 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:49.771730    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b164a3b_fc9f_48c7_a6ac_c2e3012379b8.slice/crio-31117f668e23aba8dbe0960f5402310c2944d4d8144cbf8d828901c724160504 WatchSource:0}: Error finding container 31117f668e23aba8dbe0960f5402310c2944d4d8144cbf8d828901c724160504: Status 404 returned error can't find the container with id 31117f668e23aba8dbe0960f5402310c2944d4d8144cbf8d828901c724160504
Sep 17 16:42:49.791863 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.791836    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9"]
Sep 17 16:42:49.805416 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:49.805387    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada1cced_1b84_44be_bcf1_4dc82bed86ec.slice/crio-421773ee77c3844ea99303360b4e99b3ace2b09e6944b10187abb80936273594 WatchSource:0}: Error finding container 421773ee77c3844ea99303360b4e99b3ace2b09e6944b10187abb80936273594: Status 404 returned error can't find the container with id 421773ee77c3844ea99303360b4e99b3ace2b09e6944b10187abb80936273594
Sep 17 16:42:49.866133 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:49.866098    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7ffb56c84c-bqjhr"]
Sep 17 16:42:49.868665 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:49.868641    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1f5aae_b2a9_4f09_8fef_0572bf038c72.slice/crio-82a2b210da08f57228eab4fffb102787819bfab57658380570fe1b90578f5bbb WatchSource:0}: Error finding container 82a2b210da08f57228eab4fffb102787819bfab57658380570fe1b90578f5bbb: Status 404 returned error can't find the container with id 82a2b210da08f57228eab4fffb102787819bfab57658380570fe1b90578f5bbb
Sep 17 16:42:50.216601 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:50.216527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9" event={"ID":"ada1cced-1b84-44be-bcf1-4dc82bed86ec","Type":"ContainerStarted","Data":"421773ee77c3844ea99303360b4e99b3ace2b09e6944b10187abb80936273594"}
Sep 17 16:42:50.218008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:50.217980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b7d668668-7nwdq" event={"ID":"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8","Type":"ContainerStarted","Data":"5153ca790f4c5755905dce33f970d176970b3f9eeeddf9175abf76913a374032"}
Sep 17 16:42:50.218113 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:50.218018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b7d668668-7nwdq" event={"ID":"9b164a3b-fc9f-48c7-a6ac-c2e3012379b8","Type":"ContainerStarted","Data":"31117f668e23aba8dbe0960f5402310c2944d4d8144cbf8d828901c724160504"}
Sep 17 16:42:50.219627 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:50.219603    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr" event={"ID":"dd1f5aae-b2a9-4f09-8fef-0572bf038c72","Type":"ContainerStarted","Data":"90e93508df8d9e0939cd66c753f02fd2264e77a0f9db64c144838fefd1ab3922"}
Sep 17 16:42:50.219698 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:50.219649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr" event={"ID":"dd1f5aae-b2a9-4f09-8fef-0572bf038c72","Type":"ContainerStarted","Data":"82a2b210da08f57228eab4fffb102787819bfab57658380570fe1b90578f5bbb"}
Sep 17 16:42:50.219698 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:50.219682    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:42:50.237820 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:50.237776    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-b7d668668-7nwdq" podStartSLOduration=33.23776312 podStartE2EDuration="33.23776312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:50.237110387 +0000 UTC m=+78.949117654" watchObservedRunningTime="2026-09-17 16:42:50.23776312 +0000 UTC m=+78.949770388"
Sep 17 16:42:50.255745 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:50.255696    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr" podStartSLOduration=33.255680096 podStartE2EDuration="33.255680096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:50.254582681 +0000 UTC m=+78.966589950" watchObservedRunningTime="2026-09-17 16:42:50.255680096 +0000 UTC m=+78.967687364"
Sep 17 16:42:50.497230 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:50.497189    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:50.499991 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:50.499966    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:51.187956 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.187921    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9kh8p"
Sep 17 16:42:51.223854 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.223827    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:51.225158 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.225134    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b7d668668-7nwdq"
Sep 17 16:42:51.770237 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.770206    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7ffb56c84c-bqjhr"]
Sep 17 16:42:51.802790 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.802770    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr"]
Sep 17 16:42:51.805920 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.805901    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr"
Sep 17 16:42:51.809120 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.809094    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 16:42:51.809230 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.809131    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tx27p\""
Sep 17 16:42:51.810100 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.810077    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 16:42:51.816079 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.816058    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr"]
Sep 17 16:42:51.836192 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.836156    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:51.896894 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.896867    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-674f4bb9d7-lq5lr"]
Sep 17 16:42:51.898817 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.898802    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:51.915112 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.915086    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674f4bb9d7-lq5lr"]
Sep 17 16:42:51.923150 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.923125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/42e07a4b-4c5d-4099-ad81-a7a58dccc1b8-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-fgjdr\" (UID: \"42e07a4b-4c5d-4099-ad81-a7a58dccc1b8\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr"
Sep 17 16:42:51.923309 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.923180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/42e07a4b-4c5d-4099-ad81-a7a58dccc1b8-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fgjdr\" (UID: \"42e07a4b-4c5d-4099-ad81-a7a58dccc1b8\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr"
Sep 17 16:42:51.934029 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:51.933981    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:52.024462 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.024436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/42e07a4b-4c5d-4099-ad81-a7a58dccc1b8-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-fgjdr\" (UID: \"42e07a4b-4c5d-4099-ad81-a7a58dccc1b8\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr"
Sep 17 16:42:52.024599 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.024466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/42e07a4b-4c5d-4099-ad81-a7a58dccc1b8-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fgjdr\" (UID: \"42e07a4b-4c5d-4099-ad81-a7a58dccc1b8\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr"
Sep 17 16:42:52.024599 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.024492    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wcq\" (UniqueName: \"kubernetes.io/projected/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-kube-api-access-j4wcq\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.024599 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.024527    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-trusted-ca\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.024599 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.024555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-image-registry-private-configuration\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.024599 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.024577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-registry-tls\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.024878 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.024665    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-bound-sa-token\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.024878 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.024741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-ca-trust-extracted\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.024878 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.024768    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-installation-pull-secrets\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.024878 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.024848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-registry-certificates\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.025272 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.025222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/42e07a4b-4c5d-4099-ad81-a7a58dccc1b8-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-fgjdr\" (UID: \"42e07a4b-4c5d-4099-ad81-a7a58dccc1b8\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr"
Sep 17 16:42:52.027265 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.027228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/42e07a4b-4c5d-4099-ad81-a7a58dccc1b8-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fgjdr\" (UID: \"42e07a4b-4c5d-4099-ad81-a7a58dccc1b8\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr"
Sep 17 16:42:52.029432 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.029393    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:52.125480 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.125457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wcq\" (UniqueName: \"kubernetes.io/projected/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-kube-api-access-j4wcq\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.125571 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.125491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-trusted-ca\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.125571 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.125514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-image-registry-private-configuration\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.125671 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.125564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-registry-tls\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.125671 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.125604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-bound-sa-token\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.125671 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.125633    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-ca-trust-extracted\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.125671 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.125660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-installation-pull-secrets\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.125848 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.125703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-registry-certificates\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.126134 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.126090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-ca-trust-extracted\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.126518 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.126496    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-registry-certificates\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.126682 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.126657    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-trusted-ca\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.128710 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.128153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-image-registry-private-configuration\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.128710 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.128174    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-registry-tls\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.128710 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.128659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-installation-pull-secrets\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.130047 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.130017    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:52.133192 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.133175    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wcq\" (UniqueName: \"kubernetes.io/projected/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-kube-api-access-j4wcq\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.133956 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.133935    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr"
Sep 17 16:42:52.137790 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.137771    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2e287a7-f9ee-4fe6-86c9-a987d3485c1c-bound-sa-token\") pod \"image-registry-674f4bb9d7-lq5lr\" (UID: \"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c\") " pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.207583 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.207557    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:52.230942 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.230802    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9" event={"ID":"ada1cced-1b84-44be-bcf1-4dc82bed86ec","Type":"ContainerStarted","Data":"266b0eb799cc8ab161e2299ca089165bfac4e3cf10d2198b7e41705a1f2d2abb"}
Sep 17 16:42:52.230942 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.230914    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:52.271088 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.271044    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qd8v9" podStartSLOduration=33.298508765 podStartE2EDuration="35.271027976s" totalImagesPullingTime="1.972519211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:52.270384871 +0000 UTC m=+80.982392137" watchObservedRunningTime="2026-09-17 16:42:52.271027976 +0000 UTC m=+80.983035286"
Sep 17 16:42:52.274910 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.274859    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62"]
Sep 17 16:42:52.277378 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.277355    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62"
Sep 17 16:42:52.281000 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.280978    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7zfm7\""
Sep 17 16:42:52.281349 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.281330    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 16:42:52.301349 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.301324    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62"]
Sep 17 16:42:52.356762 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.356733    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:52.405956 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.405931    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr"]
Sep 17 16:42:52.408624 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:52.408603    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e07a4b_4c5d_4099_ad81_a7a58dccc1b8.slice/crio-3e83666a5e599cf1f3cca9bee665fdd7896c462a8533ada0470c54846c8bc8a0 WatchSource:0}: Error finding container 3e83666a5e599cf1f3cca9bee665fdd7896c462a8533ada0470c54846c8bc8a0: Status 404 returned error can't find the container with id 3e83666a5e599cf1f3cca9bee665fdd7896c462a8533ada0470c54846c8bc8a0
Sep 17 16:42:52.428235 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.428216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/01ade8bd-2c86-442f-8652-1d14b67163c5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-jlh62\" (UID: \"01ade8bd-2c86-442f-8652-1d14b67163c5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62"
Sep 17 16:42:52.454505 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.454474    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:52.482515 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.482494    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674f4bb9d7-lq5lr"]
Sep 17 16:42:52.486207 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:52.486176    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e287a7_f9ee_4fe6_86c9_a987d3485c1c.slice/crio-ec83fce35b417dd86320c5bb2f3b375077346d4c93d1a32d80aa4f36782bcd6d WatchSource:0}: Error finding container ec83fce35b417dd86320c5bb2f3b375077346d4c93d1a32d80aa4f36782bcd6d: Status 404 returned error can't find the container with id ec83fce35b417dd86320c5bb2f3b375077346d4c93d1a32d80aa4f36782bcd6d
Sep 17 16:42:52.529144 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.529123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/01ade8bd-2c86-442f-8652-1d14b67163c5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-jlh62\" (UID: \"01ade8bd-2c86-442f-8652-1d14b67163c5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62"
Sep 17 16:42:52.531160 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.531137    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/01ade8bd-2c86-442f-8652-1d14b67163c5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-jlh62\" (UID: \"01ade8bd-2c86-442f-8652-1d14b67163c5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62"
Sep 17 16:42:52.586786 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.586747    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62"
Sep 17 16:42:52.854479 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:52.853465    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62"]
Sep 17 16:42:52.856129 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:52.856097    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ade8bd_2c86_442f_8652_1d14b67163c5.slice/crio-b70bebfb45f5c9e444d74e97db618fe319d39906c7c794f3a1a9a27c2bcc98fe WatchSource:0}: Error finding container b70bebfb45f5c9e444d74e97db618fe319d39906c7c794f3a1a9a27c2bcc98fe: Status 404 returned error can't find the container with id b70bebfb45f5c9e444d74e97db618fe319d39906c7c794f3a1a9a27c2bcc98fe
Sep 17 16:42:53.237109 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:53.237069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr" event={"ID":"42e07a4b-4c5d-4099-ad81-a7a58dccc1b8","Type":"ContainerStarted","Data":"3e83666a5e599cf1f3cca9bee665fdd7896c462a8533ada0470c54846c8bc8a0"}
Sep 17 16:42:53.238561 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:53.238529    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62" event={"ID":"01ade8bd-2c86-442f-8652-1d14b67163c5","Type":"ContainerStarted","Data":"b70bebfb45f5c9e444d74e97db618fe319d39906c7c794f3a1a9a27c2bcc98fe"}
Sep 17 16:42:53.240315 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:53.240285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr" event={"ID":"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c","Type":"ContainerStarted","Data":"b9c43e3438273193e86ac4cca249d793539cd6a2b9bf1b827239907f857f7f27"}
Sep 17 16:42:53.240455 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:53.240330    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr" event={"ID":"d2e287a7-f9ee-4fe6-86c9-a987d3485c1c","Type":"ContainerStarted","Data":"ec83fce35b417dd86320c5bb2f3b375077346d4c93d1a32d80aa4f36782bcd6d"}
Sep 17 16:42:53.240455 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:53.240369    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:42:53.260721 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:53.260678    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr" podStartSLOduration=2.260666763 podStartE2EDuration="2.260666763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:53.259573587 +0000 UTC m=+81.971580856" watchObservedRunningTime="2026-09-17 16:42:53.260666763 +0000 UTC m=+81.972674031"
Sep 17 16:42:53.738003 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:53.737653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:53.740580 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:53.740549    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd5495c6-27d1-4d2f-a513-52453bc60ca2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qzw8z\" (UID: \"dd5495c6-27d1-4d2f-a513-52453bc60ca2\") " pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:54.011091 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:54.011014    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-grs8h\""
Sep 17 16:42:54.018160 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:54.018128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qzw8z"
Sep 17 16:42:54.245594 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:54.245560    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr" event={"ID":"42e07a4b-4c5d-4099-ad81-a7a58dccc1b8","Type":"ContainerStarted","Data":"29fe7432972b31ba1a0717e2b32e07082c8183fa87d845eb224f0c5b06a0435b"}
Sep 17 16:42:54.247240 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:54.247214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62" event={"ID":"01ade8bd-2c86-442f-8652-1d14b67163c5","Type":"ContainerStarted","Data":"41b5163dd7eeab4c5843c7c994a67d0b9da4ec016df5ede8398194eefc17ba2d"}
Sep 17 16:42:54.261596 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:54.261529    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fgjdr" podStartSLOduration=1.568492355 podStartE2EDuration="3.261517192s" totalImagesPullingTime="1.693024837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:54.261039345 +0000 UTC m=+82.973046613" watchObservedRunningTime="2026-09-17 16:42:54.261517192 +0000 UTC m=+82.973524487"
Sep 17 16:42:54.278832 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:54.278796    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62" podStartSLOduration=1.032691007 podStartE2EDuration="2.278783243s" totalImagesPullingTime="1.246092236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:54.278146408 +0000 UTC m=+82.990153679" watchObservedRunningTime="2026-09-17 16:42:54.278783243 +0000 UTC m=+82.990790510"
Sep 17 16:42:54.369646 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:54.369624    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qzw8z"]
Sep 17 16:42:54.372101 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:54.372080    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5495c6_27d1_4d2f_a513_52453bc60ca2.slice/crio-83f387e327cab61edb5051ea77cf14b4218c216e72bc497ce6cf8b103f207426 WatchSource:0}: Error finding container 83f387e327cab61edb5051ea77cf14b4218c216e72bc497ce6cf8b103f207426: Status 404 returned error can't find the container with id 83f387e327cab61edb5051ea77cf14b4218c216e72bc497ce6cf8b103f207426
Sep 17 16:42:54.375545 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:54.375496    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:54.443789 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:54.443766    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:55.254853 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:55.254815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qzw8z" event={"ID":"dd5495c6-27d1-4d2f-a513-52453bc60ca2","Type":"ContainerStarted","Data":"435007afa2f4c5c5a6258db46e17f1c9c12e9d47699bd4a2014df0ba78c1a330"}
Sep 17 16:42:55.255298 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:55.254906    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qzw8z" event={"ID":"dd5495c6-27d1-4d2f-a513-52453bc60ca2","Type":"ContainerStarted","Data":"843b5e42ee2507c4cd7767b386649c67b93cf80b50d0fc1b9bfd12e2990cdd50"}
Sep 17 16:42:55.255298 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:55.254940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qzw8z" event={"ID":"dd5495c6-27d1-4d2f-a513-52453bc60ca2","Type":"ContainerStarted","Data":"83f387e327cab61edb5051ea77cf14b4218c216e72bc497ce6cf8b103f207426"}
Sep 17 16:42:55.255298 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:55.254974    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62"
Sep 17 16:42:55.259354 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:55.259333    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jlh62"
Sep 17 16:42:56.326736 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.326685    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:56.374485 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.374459    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"]
Sep 17 16:42:56.377702 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.377685    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.380784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.380763    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mdlt7\""
Sep 17 16:42:56.381062 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.381046    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 16:42:56.381130 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.381050    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 16:42:56.381855 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.381836    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 16:42:56.388457 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.388435    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"]
Sep 17 16:42:56.420099 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.420076    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:56.455518 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.455495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943ec8cb-97a1-4399-9f17-7860e6a22e3f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.455595 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.455532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/943ec8cb-97a1-4399-9f17-7860e6a22e3f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.455595 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.455548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl4wn\" (UniqueName: \"kubernetes.io/projected/943ec8cb-97a1-4399-9f17-7860e6a22e3f-kube-api-access-hl4wn\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.455595 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.455566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/943ec8cb-97a1-4399-9f17-7860e6a22e3f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.555888 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.555865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943ec8cb-97a1-4399-9f17-7860e6a22e3f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.555960 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.555896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/943ec8cb-97a1-4399-9f17-7860e6a22e3f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.555960 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.555913    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl4wn\" (UniqueName: \"kubernetes.io/projected/943ec8cb-97a1-4399-9f17-7860e6a22e3f-kube-api-access-hl4wn\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.555960 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.555932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/943ec8cb-97a1-4399-9f17-7860e6a22e3f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.556069 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:56.556005    2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 16:42:56.556069 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:42:56.556050    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/943ec8cb-97a1-4399-9f17-7860e6a22e3f-prometheus-operator-tls podName:943ec8cb-97a1-4399-9f17-7860e6a22e3f nodeName:}" failed. No retries permitted until 2026-09-17 16:42:57.05603614 +0000 UTC m=+85.768043386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/943ec8cb-97a1-4399-9f17-7860e6a22e3f-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-2wlcg" (UID: "943ec8cb-97a1-4399-9f17-7860e6a22e3f") : secret "prometheus-operator-tls" not found
Sep 17 16:42:56.556537 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.556518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943ec8cb-97a1-4399-9f17-7860e6a22e3f-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.558149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.558130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/943ec8cb-97a1-4399-9f17-7860e6a22e3f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.565426 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.565404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl4wn\" (UniqueName: \"kubernetes.io/projected/943ec8cb-97a1-4399-9f17-7860e6a22e3f-kube-api-access-hl4wn\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:56.637100 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:56.637046    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:57.059050 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:57.059024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/943ec8cb-97a1-4399-9f17-7860e6a22e3f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:57.061405 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:57.061384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/943ec8cb-97a1-4399-9f17-7860e6a22e3f-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-2wlcg\" (UID: \"943ec8cb-97a1-4399-9f17-7860e6a22e3f\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:57.265390 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:57.265358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qzw8z" event={"ID":"dd5495c6-27d1-4d2f-a513-52453bc60ca2","Type":"ContainerStarted","Data":"144a8520aa07b7baa31a137f949b2ec481a0c9739990629bee83f8d7baa059c7"}
Sep 17 16:42:57.283203 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:57.283163    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qzw8z" podStartSLOduration=34.397468624 podStartE2EDuration="36.283149844s" totalImagesPullingTime="1.88568122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:57.282529221 +0000 UTC m=+85.994536489" watchObservedRunningTime="2026-09-17 16:42:57.283149844 +0000 UTC m=+85.995157112"
Sep 17 16:42:57.311844 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:57.311794    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"
Sep 17 16:42:57.582559 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:57.582531    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg"]
Sep 17 16:42:57.584863 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:42:57.584838    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943ec8cb_97a1_4399_9f17_7860e6a22e3f.slice/crio-cc73705a9299e8478315ebdacc33b4b237d984f1a0dec115a76484d560d9f893 WatchSource:0}: Error finding container cc73705a9299e8478315ebdacc33b4b237d984f1a0dec115a76484d560d9f893: Status 404 returned error can't find the container with id cc73705a9299e8478315ebdacc33b4b237d984f1a0dec115a76484d560d9f893
Sep 17 16:42:58.270403 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:58.270369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg" event={"ID":"943ec8cb-97a1-4399-9f17-7860e6a22e3f","Type":"ContainerStarted","Data":"cc73705a9299e8478315ebdacc33b4b237d984f1a0dec115a76484d560d9f893"}
Sep 17 16:42:58.993631 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:58.993593    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:59.060590 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:59.060524    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:59.275876 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:59.275845    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg" event={"ID":"943ec8cb-97a1-4399-9f17-7860e6a22e3f","Type":"ContainerStarted","Data":"af7e1eb727bd09d53dae00bcc545d0b97f664f8efd5b974f115df77088c72f2d"}
Sep 17 16:42:59.276018 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:59.275895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg" event={"ID":"943ec8cb-97a1-4399-9f17-7860e6a22e3f","Type":"ContainerStarted","Data":"bd1ba90f381eb5674eec985261718d830caabd5453ce5fd973e0efa58976a387"}
Sep 17 16:42:59.295487 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:59.295424    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-2wlcg" podStartSLOduration=2.134518676 podStartE2EDuration="3.295406087s" totalImagesPullingTime="1.160887411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:59.295399801 +0000 UTC m=+88.007407075" watchObservedRunningTime="2026-09-17 16:42:59.295406087 +0000 UTC m=+88.007413355"
Sep 17 16:42:59.948592 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:42:59.948553    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:00.013407 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.013378    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:00.490771 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.490734    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:00.590009 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.589972    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:00.692083 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.692047    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:00.720813 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.720775    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"]
Sep 17 16:43:00.724149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.724132    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.726924 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.726904    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 16:43:00.727023 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.726924    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 16:43:00.727237 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.727222    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jh7h4\""
Sep 17 16:43:00.740397 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.740376    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"]
Sep 17 16:43:00.756611 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.756559    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sntj7"]
Sep 17 16:43:00.759884 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.759870    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.762157 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.762140    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 16:43:00.762339 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.762299    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tw48h\""
Sep 17 16:43:00.762426 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.762390    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 16:43:00.762669 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.762647    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 16:43:00.792355 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.792326    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:00.863981 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.863953    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:00.889953 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.889927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blgx4\" (UniqueName: \"kubernetes.io/projected/c6703ecf-d009-4e17-9503-39272491d079-kube-api-access-blgx4\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.890027 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.889956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20058c2c-35ba-4418-ad02-e28d14fa5aa7-metrics-client-ca\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.890027 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.889981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.890027 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.890023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-textfile\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.890147 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.890097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20058c2c-35ba-4418-ad02-e28d14fa5aa7-sys\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.890147 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.890124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6703ecf-d009-4e17-9503-39272491d079-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.890215 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.890155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6703ecf-d009-4e17-9503-39272491d079-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.890215 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.890180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-tls\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.890215 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.890209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20058c2c-35ba-4418-ad02-e28d14fa5aa7-root\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.890360 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.890248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6703ecf-d009-4e17-9503-39272491d079-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.890360 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.890291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.890360 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.890312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fq5c\" (UniqueName: \"kubernetes.io/projected/20058c2c-35ba-4418-ad02-e28d14fa5aa7-kube-api-access-6fq5c\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.890360 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.890342    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-wtmp\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.960170 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.960144    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:00.991194 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6703ecf-d009-4e17-9503-39272491d079-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.991294 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991203    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6703ecf-d009-4e17-9503-39272491d079-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.991294 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-tls\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.991294 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20058c2c-35ba-4418-ad02-e28d14fa5aa7-root\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.991435 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6703ecf-d009-4e17-9503-39272491d079-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.991435 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20058c2c-35ba-4418-ad02-e28d14fa5aa7-root\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.991526 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.991567 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fq5c\" (UniqueName: \"kubernetes.io/projected/20058c2c-35ba-4418-ad02-e28d14fa5aa7-kube-api-access-6fq5c\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.991618 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-wtmp\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.991689 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgx4\" (UniqueName: \"kubernetes.io/projected/c6703ecf-d009-4e17-9503-39272491d079-kube-api-access-blgx4\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.991743 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20058c2c-35ba-4418-ad02-e28d14fa5aa7-metrics-client-ca\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.991795 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.991795 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-textfile\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.991894 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20058c2c-35ba-4418-ad02-e28d14fa5aa7-sys\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.991949 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20058c2c-35ba-4418-ad02-e28d14fa5aa7-sys\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.992002 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.991986    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6703ecf-d009-4e17-9503-39272491d079-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.992198 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.992121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.992198 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.992129    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-wtmp\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.992365 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.992326    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-textfile\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.992488 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.992469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20058c2c-35ba-4418-ad02-e28d14fa5aa7-metrics-client-ca\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.993784 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.993764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-tls\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:00.993857 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.993783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6703ecf-d009-4e17-9503-39272491d079-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.994076 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.994055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6703ecf-d009-4e17-9503-39272491d079-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:00.994198 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:00.994184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20058c2c-35ba-4418-ad02-e28d14fa5aa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:01.002526 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.002504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fq5c\" (UniqueName: \"kubernetes.io/projected/20058c2c-35ba-4418-ad02-e28d14fa5aa7-kube-api-access-6fq5c\") pod \"node-exporter-sntj7\" (UID: \"20058c2c-35ba-4418-ad02-e28d14fa5aa7\") " pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:01.002941 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.002921    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgx4\" (UniqueName: \"kubernetes.io/projected/c6703ecf-d009-4e17-9503-39272491d079-kube-api-access-blgx4\") pod \"openshift-state-metrics-86c5bc79d6-hllw6\" (UID: \"c6703ecf-d009-4e17-9503-39272491d079\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:01.019437 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.019387    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:01.032858 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.032841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"
Sep 17 16:43:01.068892 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.068868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sntj7"
Sep 17 16:43:01.087007 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:43:01.086957    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20058c2c_35ba_4418_ad02_e28d14fa5aa7.slice/crio-ccb08a47f7655572ba56fa2baa0cc51432d4d843212f1aeecef485762a071cc8 WatchSource:0}: Error finding container ccb08a47f7655572ba56fa2baa0cc51432d4d843212f1aeecef485762a071cc8: Status 404 returned error can't find the container with id ccb08a47f7655572ba56fa2baa0cc51432d4d843212f1aeecef485762a071cc8
Sep 17 16:43:01.235716 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.235683    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:01.283886 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.283821    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sntj7" event={"ID":"20058c2c-35ba-4418-ad02-e28d14fa5aa7","Type":"ContainerStarted","Data":"ccb08a47f7655572ba56fa2baa0cc51432d4d843212f1aeecef485762a071cc8"}
Sep 17 16:43:01.308031 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.308005    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6"]
Sep 17 16:43:01.312393 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:43:01.312372    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6703ecf_d009_4e17_9503_39272491d079.slice/crio-0f4dfc2362445cab7e63810b222308cbffd63598dda9fbfcc78c97b558ecab49 WatchSource:0}: Error finding container 0f4dfc2362445cab7e63810b222308cbffd63598dda9fbfcc78c97b558ecab49: Status 404 returned error can't find the container with id 0f4dfc2362445cab7e63810b222308cbffd63598dda9fbfcc78c97b558ecab49
Sep 17 16:43:01.699159 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.699066    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:01.797563 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.797164    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:43:01.801604 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.801580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.804622 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.804343    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 16:43:01.804622 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.804402    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8k6kh\""
Sep 17 16:43:01.804622 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.804413    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 16:43:01.804622 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.804343    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 16:43:01.805039 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.804824    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 16:43:01.805039 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.804843    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 16:43:01.805322 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.805305    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 16:43:01.805633 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.805611    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 16:43:01.807216 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.807049    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 16:43:01.807216 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.807053    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 16:43:01.818191 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.818167    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:43:01.829858 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.829814    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:01.898976 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.898940    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899144 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.898981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79hff\" (UniqueName: \"kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-kube-api-access-79hff\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899144 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.899053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899144 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.899085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899144 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.899113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899358 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.899145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-web-config\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899358 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.899192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-config-out\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899358 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.899214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899358 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.899278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899358 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.899297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899972 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.899842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899972 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.899875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:01.899972 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:01.899896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001384 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001544 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-web-config\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001627 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001530    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-config-out\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001627 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001735 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001735 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001735 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001735 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001918 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001756    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001918 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001777    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001918 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001918 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79hff\" (UniqueName: \"kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-kube-api-access-79hff\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001918 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.001918 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.001916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.003186 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.002843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.003376 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:43:02.003355    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-trusted-ca-bundle podName:dc855443-4820-4b92-8abc-0afaed1cf7da nodeName:}" failed. No retries permitted until 2026-09-17 16:43:02.503333393 +0000 UTC m=+91.215340647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da") : configmap references non-existent config key: ca-bundle.crt
Sep 17 16:43:02.006735 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.006687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.006933 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.006908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.008730 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.008704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.008828 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.008760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-web-config\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.008989 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.008942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-config-out\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.010123 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.010101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.010273 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.010188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.011164 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.011140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.011287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.011150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.013327 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.013287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79hff\" (UniqueName: \"kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-kube-api-access-79hff\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.108580 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.108536    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:02.290493 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.290421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6" event={"ID":"c6703ecf-d009-4e17-9503-39272491d079","Type":"ContainerStarted","Data":"8ff2bfe7a862a689a9bbf5b1014dbf0d1413bef29eea93c2826da0afdea14a73"}
Sep 17 16:43:02.290634 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.290506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6" event={"ID":"c6703ecf-d009-4e17-9503-39272491d079","Type":"ContainerStarted","Data":"54ec982da7757db4597bfab41bd2125b2eea8215df4631d40194ee4dfab6f6af"}
Sep 17 16:43:02.290634 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.290527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6" event={"ID":"c6703ecf-d009-4e17-9503-39272491d079","Type":"ContainerStarted","Data":"0f4dfc2362445cab7e63810b222308cbffd63598dda9fbfcc78c97b558ecab49"}
Sep 17 16:43:02.506885 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.506848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.507739 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.507717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.714414 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.714384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:02.950852 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:02.950783    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:03.020235 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.020206    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:43:03.022012 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:43:03.021987    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc855443_4820_4b92_8abc_0afaed1cf7da.slice/crio-9d73b58b52d5dde89b35a883304fc612ddb689dd9d5a417b2c51443c0a5c0d47 WatchSource:0}: Error finding container 9d73b58b52d5dde89b35a883304fc612ddb689dd9d5a417b2c51443c0a5c0d47: Status 404 returned error can't find the container with id 9d73b58b52d5dde89b35a883304fc612ddb689dd9d5a417b2c51443c0a5c0d47
Sep 17 16:43:03.049504 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.049474    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:03.212175 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.212116    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:03.295639 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.295610    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerStarted","Data":"9d73b58b52d5dde89b35a883304fc612ddb689dd9d5a417b2c51443c0a5c0d47"}
Sep 17 16:43:03.297287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.297247    2566 generic.go:356] "Generic (PLEG): container finished" podID="20058c2c-35ba-4418-ad02-e28d14fa5aa7" containerID="020bf0c003a36a1c2b0133694ebfc01292fdb78e02bf067e605db266270e6b21" exitCode=0
Sep 17 16:43:03.297384 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.297283    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sntj7" event={"ID":"20058c2c-35ba-4418-ad02-e28d14fa5aa7","Type":"ContainerDied","Data":"020bf0c003a36a1c2b0133694ebfc01292fdb78e02bf067e605db266270e6b21"}
Sep 17 16:43:03.299684 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.299657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6" event={"ID":"c6703ecf-d009-4e17-9503-39272491d079","Type":"ContainerStarted","Data":"4c79ec5f01c17d4a1aad2c167af98b18354fd6e1e0ba130206ac5dc2d5378c46"}
Sep 17 16:43:03.339777 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.339712    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hllw6" podStartSLOduration=2.254953635 podStartE2EDuration="3.339698102s" totalImagesPullingTime="1.084744467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:03.339018807 +0000 UTC m=+92.051026076" watchObservedRunningTime="2026-09-17 16:43:03.339698102 +0000 UTC m=+92.051705370"
Sep 17 16:43:03.584026 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.584001    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:03.682593 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.682559    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:03.779531 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.779504    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:03.847752 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.847566    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:03.980038 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:03.980001    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:04.081425 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:04.081385    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:04.180632 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:04.180553    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:04.280758 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:04.280721    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:04.306160 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:04.306127    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sntj7" event={"ID":"20058c2c-35ba-4418-ad02-e28d14fa5aa7","Type":"ContainerStarted","Data":"163ae832fb08eccfb2f5ccc6293934979f4bae9f6250aaf86d0155d43b895e79"}
Sep 17 16:43:04.306323 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:04.306246    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sntj7" event={"ID":"20058c2c-35ba-4418-ad02-e28d14fa5aa7","Type":"ContainerStarted","Data":"44429f410af96f078dd87516074805e38a716d7c4d5879482b4570b5b0753e58"}
Sep 17 16:43:04.331135 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:04.331095    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sntj7" podStartSLOduration=3.114910583 podStartE2EDuration="4.331081007s" totalImagesPullingTime="1.145747544s" totalInitContainerRuntime="70.42288ms" isStatefulPod=true podCreationTimestamp="2026-09-17 16:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:04.330634474 +0000 UTC m=+93.042641763" watchObservedRunningTime="2026-09-17 16:43:04.331081007 +0000 UTC m=+93.043088275"
Sep 17 16:43:04.405508 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:04.405481    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:04.600261 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:04.600214    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:04.861932 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:04.861906    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:05.313355 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:05.313324    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerID="b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579" exitCode=0
Sep 17 16:43:05.313753 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:05.313409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerDied","Data":"b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579"}
Sep 17 16:43:05.495339 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:05.495307    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:05.595847 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:05.595773    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:06.014268 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:06.014220    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:06.220741 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:06.220706    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:06.436944 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:06.436869    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:07.075488 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:07.075453    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:07.255162 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:07.255124    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:07.353682 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:07.353557    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:08.139098 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:08.139025    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:08.194150 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:08.194116    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:08.328826 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:08.328803    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerStarted","Data":"c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2"}
Sep 17 16:43:08.328924 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:08.328884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerStarted","Data":"671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3"}
Sep 17 16:43:08.328924 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:08.328913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerStarted","Data":"a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069"}
Sep 17 16:43:08.328994 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:08.328936    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerStarted","Data":"211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9"}
Sep 17 16:43:08.328994 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:08.328959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerStarted","Data":"07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e"}
Sep 17 16:43:09.336922 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:09.336890    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerStarted","Data":"72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307"}
Sep 17 16:43:09.366346 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:09.366300    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.964405905 podStartE2EDuration="8.366285581s" totalImagesPullingTime="4.383329242s" totalInitContainerRuntime="18.550434ms" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:09.364138577 +0000 UTC m=+98.076145856" watchObservedRunningTime="2026-09-17 16:43:09.366285581 +0000 UTC m=+98.078292850"
Sep 17 16:43:10.923345 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:10.923312    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:10.982964 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:10.982936    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:12.236622 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:12.236595    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:43:14.252000 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:14.251968    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-674f4bb9d7-lq5lr"
Sep 17 16:43:14.398272 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:14.398226    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:14.507244 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:14.507173    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:15.202693 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:15.202666    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w6lnl"
Sep 17 16:43:17.250355 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.250310    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr" podUID="dd1f5aae-b2a9-4f09-8fef-0572bf038c72" containerName="registry" containerID="cri-o://90e93508df8d9e0939cd66c753f02fd2264e77a0f9db64c144838fefd1ab3922" gracePeriod=30
Sep 17 16:43:17.370576 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.370543    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd1f5aae-b2a9-4f09-8fef-0572bf038c72" containerID="90e93508df8d9e0939cd66c753f02fd2264e77a0f9db64c144838fefd1ab3922" exitCode=0
Sep 17 16:43:17.370689 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.370571    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr" event={"ID":"dd1f5aae-b2a9-4f09-8fef-0572bf038c72","Type":"ContainerDied","Data":"90e93508df8d9e0939cd66c753f02fd2264e77a0f9db64c144838fefd1ab3922"}
Sep 17 16:43:17.494146 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.494123    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:43:17.626518 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.626460    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-ca-trust-extracted\") pod \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") "
Sep 17 16:43:17.626518 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.626505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-image-registry-private-configuration\") pod \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") "
Sep 17 16:43:17.626662 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.626524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls\") pod \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") "
Sep 17 16:43:17.626662 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.626573    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kz9k\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-kube-api-access-7kz9k\") pod \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") "
Sep 17 16:43:17.626662 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.626595    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-installation-pull-secrets\") pod \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") "
Sep 17 16:43:17.626790 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.626660    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-certificates\") pod \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") "
Sep 17 16:43:17.626790 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.626687    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-bound-sa-token\") pod \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") "
Sep 17 16:43:17.626790 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.626712    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-trusted-ca\") pod \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\" (UID: \"dd1f5aae-b2a9-4f09-8fef-0572bf038c72\") "
Sep 17 16:43:17.627304 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.627098    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dd1f5aae-b2a9-4f09-8fef-0572bf038c72" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:17.627407 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.627349    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dd1f5aae-b2a9-4f09-8fef-0572bf038c72" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:17.629141 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.629108    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-kube-api-access-7kz9k" (OuterVolumeSpecName: "kube-api-access-7kz9k") pod "dd1f5aae-b2a9-4f09-8fef-0572bf038c72" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72"). InnerVolumeSpecName "kube-api-access-7kz9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:17.629273 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.629201    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dd1f5aae-b2a9-4f09-8fef-0572bf038c72" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:17.629273 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.629209    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dd1f5aae-b2a9-4f09-8fef-0572bf038c72" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:17.629366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.629287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dd1f5aae-b2a9-4f09-8fef-0572bf038c72" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:17.629487 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.629464    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dd1f5aae-b2a9-4f09-8fef-0572bf038c72" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:17.635110 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.635076    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dd1f5aae-b2a9-4f09-8fef-0572bf038c72" (UID: "dd1f5aae-b2a9-4f09-8fef-0572bf038c72"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:43:17.727544 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.727516    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-image-registry-private-configuration\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:43:17.727544 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.727538    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:43:17.727667 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.727549    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kz9k\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-kube-api-access-7kz9k\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:43:17.727667 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.727558    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-installation-pull-secrets\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:43:17.727667 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.727567    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-registry-certificates\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:43:17.727667 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.727575    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-bound-sa-token\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:43:17.727667 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.727583    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-trusted-ca\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:43:17.727667 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:17.727590    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd1f5aae-b2a9-4f09-8fef-0572bf038c72-ca-trust-extracted\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:43:18.376058 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:18.376026    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr" event={"ID":"dd1f5aae-b2a9-4f09-8fef-0572bf038c72","Type":"ContainerDied","Data":"82a2b210da08f57228eab4fffb102787819bfab57658380570fe1b90578f5bbb"}
Sep 17 16:43:18.376058 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:18.376052    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ffb56c84c-bqjhr"
Sep 17 16:43:18.376584 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:18.376095    2566 scope.go:117] "RemoveContainer" containerID="90e93508df8d9e0939cd66c753f02fd2264e77a0f9db64c144838fefd1ab3922"
Sep 17 16:43:18.394219 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:18.394194    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7ffb56c84c-bqjhr"]
Sep 17 16:43:18.400076 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:18.400055    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7ffb56c84c-bqjhr"]
Sep 17 16:43:18.610267 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:18.610205    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:18.673451 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:18.673401    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:19.810802 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:19.810769    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd1f5aae-b2a9-4f09-8fef-0572bf038c72" path="/var/lib/kubelet/pods/dd1f5aae-b2a9-4f09-8fef-0572bf038c72/volumes"
Sep 17 16:43:21.007688 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.007657    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c9bd9f99f-f9grr"]
Sep 17 16:43:21.008037 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.007986    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd1f5aae-b2a9-4f09-8fef-0572bf038c72" containerName="registry"
Sep 17 16:43:21.008037 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.007998    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd1f5aae-b2a9-4f09-8fef-0572bf038c72" containerName="registry"
Sep 17 16:43:21.008104 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.008056    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd1f5aae-b2a9-4f09-8fef-0572bf038c72" containerName="registry"
Sep 17 16:43:21.011329 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.011308    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.014045 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.014021    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 16:43:21.015060 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.015013    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 16:43:21.015271 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.015204    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 16:43:21.015362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.015308    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 16:43:21.015362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.015337    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 16:43:21.015362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.015311    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 16:43:21.015611 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.015596    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mbrl8\""
Sep 17 16:43:21.016543 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.016526    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 16:43:21.019982 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.019961    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 16:43:21.021136 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.021116    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9bd9f99f-f9grr"]
Sep 17 16:43:21.145596 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.145572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:21.152436 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.152414    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-trusted-ca-bundle\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.152514 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.152456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-oauth-config\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.152514 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.152481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-config\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.152514 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.152503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-oauth-serving-cert\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.152612 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.152567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-service-ca\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.152612 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.152607    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zmn\" (UniqueName: \"kubernetes.io/projected/a3bae121-5933-4f4e-870c-27be5f2c5c5d-kube-api-access-74zmn\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.152674 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.152634    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-serving-cert\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.246730 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.246708    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:21.253682 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.253661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-config\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.253756 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.253686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-oauth-serving-cert\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.253756 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.253715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-service-ca\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.253860 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.253792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zmn\" (UniqueName: \"kubernetes.io/projected/a3bae121-5933-4f4e-870c-27be5f2c5c5d-kube-api-access-74zmn\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.253860 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.253850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-serving-cert\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.253957 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.253882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-trusted-ca-bundle\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.253957 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.253938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-oauth-config\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.254508 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.254483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-service-ca\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.254602 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.254483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-oauth-serving-cert\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.254691 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.254659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-config\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.254934 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.254916    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-trusted-ca-bundle\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.256232 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.256201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-oauth-config\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.256344 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.256327    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-serving-cert\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.263311 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.263239    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zmn\" (UniqueName: \"kubernetes.io/projected/a3bae121-5933-4f4e-870c-27be5f2c5c5d-kube-api-access-74zmn\") pod \"console-7c9bd9f99f-f9grr\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") " pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.321107 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.321082    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:21.591931 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.591805    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9bd9f99f-f9grr"]
Sep 17 16:43:21.594309 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:43:21.594282    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bae121_5933_4f4e_870c_27be5f2c5c5d.slice/crio-0e002177a5fb07def9db93dbecfa672efeeab4a9f5fba0382d1efa6da9320959 WatchSource:0}: Error finding container 0e002177a5fb07def9db93dbecfa672efeeab4a9f5fba0382d1efa6da9320959: Status 404 returned error can't find the container with id 0e002177a5fb07def9db93dbecfa672efeeab4a9f5fba0382d1efa6da9320959
Sep 17 16:43:21.821150 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.821125    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:21.921003 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:21.920933    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:22.026086 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:22.026050    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:22.400109 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:22.400073    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd9f99f-f9grr" event={"ID":"a3bae121-5933-4f4e-870c-27be5f2c5c5d","Type":"ContainerStarted","Data":"0e002177a5fb07def9db93dbecfa672efeeab4a9f5fba0382d1efa6da9320959"}
Sep 17 16:43:25.925579 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:25.925546    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:26.039219 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:26.039184    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:26.126362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:26.126326    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:26.226845 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:26.226778    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:26.316754 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:26.316713    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:26.417379 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:26.417352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd9f99f-f9grr" event={"ID":"a3bae121-5933-4f4e-870c-27be5f2c5c5d","Type":"ContainerStarted","Data":"bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f"}
Sep 17 16:43:26.439667 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:26.439624    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c9bd9f99f-f9grr" podStartSLOduration=2.569273196 podStartE2EDuration="6.439611638s" totalImagesPullingTime="3.870338442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:26.438405333 +0000 UTC m=+115.150412601" watchObservedRunningTime="2026-09-17 16:43:26.439611638 +0000 UTC m=+115.151618906"
Sep 17 16:43:27.422874 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:27.422844    2566 generic.go:356] "Generic (PLEG): container finished" podID="3efa2ff4-6070-46a8-941d-0c42ee9f24e1" containerID="7130158ca670ccd99b8cb6fee34913465aa0717e1e931114c347c0b2ba79153a" exitCode=0
Sep 17 16:43:27.423233 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:27.422930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9" event={"ID":"3efa2ff4-6070-46a8-941d-0c42ee9f24e1","Type":"ContainerDied","Data":"7130158ca670ccd99b8cb6fee34913465aa0717e1e931114c347c0b2ba79153a"}
Sep 17 16:43:27.423316 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:27.423301    2566 scope.go:117] "RemoveContainer" containerID="7130158ca670ccd99b8cb6fee34913465aa0717e1e931114c347c0b2ba79153a"
Sep 17 16:43:27.820316 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:27.820280    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:27.888986 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:27.888943    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:28.428547 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:28.428518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bl8n9" event={"ID":"3efa2ff4-6070-46a8-941d-0c42ee9f24e1","Type":"ContainerStarted","Data":"a786cec7a836a1474d82c469379646b609847c8bfd015e18801eb4beec443fe8"}
Sep 17 16:43:31.322116 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:31.322066    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:31.322504 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:31.322133    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:31.326845 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:31.326820    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:31.443350 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:31.443325    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:43:31.605463 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:31.605384    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:31.705472 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:31.705439    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:31.814072 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:31.814043    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:35.153931 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:35.153899    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:35.223362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:35.223338    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:40.925785 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:40.925750    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:40.995906 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:40.995876    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:41.606447 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:41.606416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:43:41.608654 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:41.608623    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efac150-5db6-42c1-8309-138120e7f820-metrics-certs\") pod \"network-metrics-daemon-vm27c\" (UID: \"4efac150-5db6-42c1-8309-138120e7f820\") " pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:43:41.619044 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:41.619023    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lng7\""
Sep 17 16:43:41.626950 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:41.626936    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vm27c"
Sep 17 16:43:41.898887 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:41.898837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vm27c"]
Sep 17 16:43:41.901511 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:43:41.901485    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efac150_5db6_42c1_8309_138120e7f820.slice/crio-51e29ad46060a67668669d3aff4f8595cfda1d3dbeeba0005c96c1ebdd390523 WatchSource:0}: Error finding container 51e29ad46060a67668669d3aff4f8595cfda1d3dbeeba0005c96c1ebdd390523: Status 404 returned error can't find the container with id 51e29ad46060a67668669d3aff4f8595cfda1d3dbeeba0005c96c1ebdd390523
Sep 17 16:43:42.481872 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:42.481827    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vm27c" event={"ID":"4efac150-5db6-42c1-8309-138120e7f820","Type":"ContainerStarted","Data":"51e29ad46060a67668669d3aff4f8595cfda1d3dbeeba0005c96c1ebdd390523"}
Sep 17 16:43:42.483426 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:42.483401    2566 generic.go:356] "Generic (PLEG): container finished" podID="80a022e2-e6f2-4047-9d8c-83b38fdece2a" containerID="ef866c52b2877017c6cbc049e4259e55d0d9bceb88ebb5a9be0dc359fe3ff109" exitCode=0
Sep 17 16:43:42.483504 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:42.483481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27" event={"ID":"80a022e2-e6f2-4047-9d8c-83b38fdece2a","Type":"ContainerDied","Data":"ef866c52b2877017c6cbc049e4259e55d0d9bceb88ebb5a9be0dc359fe3ff109"}
Sep 17 16:43:42.483796 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:42.483782    2566 scope.go:117] "RemoveContainer" containerID="ef866c52b2877017c6cbc049e4259e55d0d9bceb88ebb5a9be0dc359fe3ff109"
Sep 17 16:43:43.488712 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:43.488674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vm27c" event={"ID":"4efac150-5db6-42c1-8309-138120e7f820","Type":"ContainerStarted","Data":"cc0a0813c5f82d6cf1b76fa6ca17b2497bf5eb936f10bdf27c65bdf780b9b5ac"}
Sep 17 16:43:43.489096 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:43.488770    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vm27c" event={"ID":"4efac150-5db6-42c1-8309-138120e7f820","Type":"ContainerStarted","Data":"820d0c2e686e6c03095b98d6fc7147ccdcff5d32a9a6cad3e1481038e84a3bd6"}
Sep 17 16:43:43.490502 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:43.490477    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zgv27" event={"ID":"80a022e2-e6f2-4047-9d8c-83b38fdece2a","Type":"ContainerStarted","Data":"2f3f5d8c8b0ae7c32a3a930d2a6705cf7050060d421a587aff3b0b7a9029e5f8"}
Sep 17 16:43:43.509551 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:43.509509    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vm27c" podStartSLOduration=130.263199306 podStartE2EDuration="2m11.509495286s" totalImagesPullingTime="1.24629598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:43.507285466 +0000 UTC m=+132.219292736" watchObservedRunningTime="2026-09-17 16:43:43.509495286 +0000 UTC m=+132.221502553"
Sep 17 16:43:43.608381 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:43.608352    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:43.663557 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:43:43.663527    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:20.054492 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:20.054453    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:20.154788 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:20.154747    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:21.123129 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.123094    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:44:21.123530 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.123384    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="alertmanager" containerID="cri-o://07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e" gracePeriod=120
Sep 17 16:44:21.123530 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.123455    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy-metric" containerID="cri-o://c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2" gracePeriod=120
Sep 17 16:44:21.123530 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.123498    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="config-reloader" containerID="cri-o://211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9" gracePeriod=120
Sep 17 16:44:21.123681 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.123530    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="prom-label-proxy" containerID="cri-o://72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307" gracePeriod=120
Sep 17 16:44:21.123681 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.123474    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy-web" containerID="cri-o://a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069" gracePeriod=120
Sep 17 16:44:21.123681 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.123518    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy" containerID="cri-o://671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3" gracePeriod=120
Sep 17 16:44:21.638677 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.638652    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerID="72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307" exitCode=0
Sep 17 16:44:21.638677 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.638675    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerID="671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3" exitCode=0
Sep 17 16:44:21.638850 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.638683    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerID="211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9" exitCode=0
Sep 17 16:44:21.638850 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.638689    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerID="07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e" exitCode=0
Sep 17 16:44:21.638850 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.638726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerDied","Data":"72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307"}
Sep 17 16:44:21.638850 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.638802    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerDied","Data":"671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3"}
Sep 17 16:44:21.638850 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.638831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerDied","Data":"211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9"}
Sep 17 16:44:21.639020 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:21.638857    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerDied","Data":"07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e"}
Sep 17 16:44:22.369519 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.369497    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.507108 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507079    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79hff\" (UniqueName: \"kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-kube-api-access-79hff\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507115    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-config-out\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507147    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-main-tls\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507171    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-config-volume\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507195    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-metrics-client-ca\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507233    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-web\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507571 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507294    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-trusted-ca-bundle\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507571 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507316    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-tls-assets\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507571 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507353    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507571 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507397    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-main-db\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507571 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507428    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507571 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507468    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-cluster-tls-config\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507571 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-web-config\") pod \"dc855443-4820-4b92-8abc-0afaed1cf7da\" (UID: \"dc855443-4820-4b92-8abc-0afaed1cf7da\") "
Sep 17 16:44:22.507960 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:22.507960 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.507783    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-metrics-client-ca\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.508595 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.508132    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:22.508595 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.508533    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:22.510105 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.510075    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-config-out" (OuterVolumeSpecName: "config-out") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:22.510105 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.510090    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-kube-api-access-79hff" (OuterVolumeSpecName: "kube-api-access-79hff") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "kube-api-access-79hff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:22.510380 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.510353    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:22.510874 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.510830    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:22.511169 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.511146    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:22.511275 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.511207    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:22.511328 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.511282    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-config-volume" (OuterVolumeSpecName: "config-volume") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:22.511912 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.511893    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:22.514647 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.514628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:22.521115 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.521093    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-web-config" (OuterVolumeSpecName: "web-config") pod "dc855443-4820-4b92-8abc-0afaed1cf7da" (UID: "dc855443-4820-4b92-8abc-0afaed1cf7da"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:22.608478 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608454    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.608478 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608476    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.608624 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608486    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-tls-assets\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.608624 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608495    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.608624 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608504    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-alertmanager-main-db\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.608624 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608513    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.608624 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608521    2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-cluster-tls-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.608624 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608529    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-web-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.608624 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608536    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79hff\" (UniqueName: \"kubernetes.io/projected/dc855443-4820-4b92-8abc-0afaed1cf7da-kube-api-access-79hff\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.608624 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608543    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc855443-4820-4b92-8abc-0afaed1cf7da-config-out\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.608624 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608550    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-secret-alertmanager-main-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.608624 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.608558    2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc855443-4820-4b92-8abc-0afaed1cf7da-config-volume\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:44:22.650716 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.650685    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerID="c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2" exitCode=0
Sep 17 16:44:22.650716 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.650712    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerID="a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069" exitCode=0
Sep 17 16:44:22.650848 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.650757    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerDied","Data":"c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2"}
Sep 17 16:44:22.650848 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.650786    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.650911 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.650849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerDied","Data":"a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069"}
Sep 17 16:44:22.650911 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.650878    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc855443-4820-4b92-8abc-0afaed1cf7da","Type":"ContainerDied","Data":"9d73b58b52d5dde89b35a883304fc612ddb689dd9d5a417b2c51443c0a5c0d47"}
Sep 17 16:44:22.650911 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.650908    2566 scope.go:117] "RemoveContainer" containerID="72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307"
Sep 17 16:44:22.658392 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.658281    2566 scope.go:117] "RemoveContainer" containerID="c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2"
Sep 17 16:44:22.665856 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.665838    2566 scope.go:117] "RemoveContainer" containerID="671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3"
Sep 17 16:44:22.672008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.671992    2566 scope.go:117] "RemoveContainer" containerID="a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069"
Sep 17 16:44:22.675581 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.675561    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:44:22.678946 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.678927    2566 scope.go:117] "RemoveContainer" containerID="211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9"
Sep 17 16:44:22.679217 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.679198    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:44:22.685404 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.685385    2566 scope.go:117] "RemoveContainer" containerID="07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e"
Sep 17 16:44:22.691946 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.691930    2566 scope.go:117] "RemoveContainer" containerID="b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579"
Sep 17 16:44:22.698343 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.698328    2566 scope.go:117] "RemoveContainer" containerID="72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307"
Sep 17 16:44:22.698595 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:44:22.698577    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307\": container with ID starting with 72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307 not found: ID does not exist" containerID="72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307"
Sep 17 16:44:22.698644 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.698603    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307"} err="failed to get container status \"72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307\": rpc error: code = NotFound desc = could not find container \"72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307\": container with ID starting with 72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307 not found: ID does not exist"
Sep 17 16:44:22.698644 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.698629    2566 scope.go:117] "RemoveContainer" containerID="c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2"
Sep 17 16:44:22.698853 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:44:22.698838    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2\": container with ID starting with c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2 not found: ID does not exist" containerID="c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2"
Sep 17 16:44:22.698889 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.698857    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2"} err="failed to get container status \"c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2\": rpc error: code = NotFound desc = could not find container \"c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2\": container with ID starting with c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2 not found: ID does not exist"
Sep 17 16:44:22.698889 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.698869    2566 scope.go:117] "RemoveContainer" containerID="671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3"
Sep 17 16:44:22.699049 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:44:22.699036    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3\": container with ID starting with 671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3 not found: ID does not exist" containerID="671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3"
Sep 17 16:44:22.699084 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.699052    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3"} err="failed to get container status \"671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3\": rpc error: code = NotFound desc = could not find container \"671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3\": container with ID starting with 671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3 not found: ID does not exist"
Sep 17 16:44:22.699084 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.699062    2566 scope.go:117] "RemoveContainer" containerID="a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069"
Sep 17 16:44:22.699242 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:44:22.699227    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069\": container with ID starting with a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069 not found: ID does not exist" containerID="a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069"
Sep 17 16:44:22.699346 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.699244    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069"} err="failed to get container status \"a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069\": rpc error: code = NotFound desc = could not find container \"a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069\": container with ID starting with a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069 not found: ID does not exist"
Sep 17 16:44:22.699385 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.699347    2566 scope.go:117] "RemoveContainer" containerID="211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9"
Sep 17 16:44:22.699582 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:44:22.699565    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9\": container with ID starting with 211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9 not found: ID does not exist" containerID="211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9"
Sep 17 16:44:22.699627 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.699585    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9"} err="failed to get container status \"211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9\": rpc error: code = NotFound desc = could not find container \"211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9\": container with ID starting with 211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9 not found: ID does not exist"
Sep 17 16:44:22.699627 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.699598    2566 scope.go:117] "RemoveContainer" containerID="07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e"
Sep 17 16:44:22.699788 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:44:22.699774    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e\": container with ID starting with 07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e not found: ID does not exist" containerID="07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e"
Sep 17 16:44:22.699826 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.699790    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e"} err="failed to get container status \"07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e\": rpc error: code = NotFound desc = could not find container \"07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e\": container with ID starting with 07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e not found: ID does not exist"
Sep 17 16:44:22.699826 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.699802    2566 scope.go:117] "RemoveContainer" containerID="b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579"
Sep 17 16:44:22.700026 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:44:22.700011    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579\": container with ID starting with b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579 not found: ID does not exist" containerID="b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579"
Sep 17 16:44:22.700075 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.700026    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579"} err="failed to get container status \"b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579\": rpc error: code = NotFound desc = could not find container \"b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579\": container with ID starting with b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579 not found: ID does not exist"
Sep 17 16:44:22.700075 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.700037    2566 scope.go:117] "RemoveContainer" containerID="72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307"
Sep 17 16:44:22.700225 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.700210    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307"} err="failed to get container status \"72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307\": rpc error: code = NotFound desc = could not find container \"72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307\": container with ID starting with 72f348140fad76760bbde0ac708bf6d25568f5e0854e9532e89efba203bcb307 not found: ID does not exist"
Sep 17 16:44:22.700290 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.700225    2566 scope.go:117] "RemoveContainer" containerID="c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2"
Sep 17 16:44:22.700431 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.700415    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2"} err="failed to get container status \"c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2\": rpc error: code = NotFound desc = could not find container \"c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2\": container with ID starting with c87a0d812575bff3d1f3782ae41e663f358cb1f5f2e6f959892c705a08f9adf2 not found: ID does not exist"
Sep 17 16:44:22.700477 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.700431    2566 scope.go:117] "RemoveContainer" containerID="671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3"
Sep 17 16:44:22.700646 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.700631    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3"} err="failed to get container status \"671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3\": rpc error: code = NotFound desc = could not find container \"671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3\": container with ID starting with 671de622d314cd3004404850a96727b72ab804c487c3c0d6ea8ac83114cd64b3 not found: ID does not exist"
Sep 17 16:44:22.700646 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.700645    2566 scope.go:117] "RemoveContainer" containerID="a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069"
Sep 17 16:44:22.700851 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.700832    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069"} err="failed to get container status \"a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069\": rpc error: code = NotFound desc = could not find container \"a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069\": container with ID starting with a13bf969645a18a6120c010bd36dddf765e514c4e5577f6b037c0616f6236069 not found: ID does not exist"
Sep 17 16:44:22.700851 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.700849    2566 scope.go:117] "RemoveContainer" containerID="211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9"
Sep 17 16:44:22.701034 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.701018    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9"} err="failed to get container status \"211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9\": rpc error: code = NotFound desc = could not find container \"211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9\": container with ID starting with 211bac1ebcb92444b426c1d7e6846b34df7b41213aa9933fb0920035c05c45f9 not found: ID does not exist"
Sep 17 16:44:22.701034 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.701033    2566 scope.go:117] "RemoveContainer" containerID="07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e"
Sep 17 16:44:22.701233 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.701219    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e"} err="failed to get container status \"07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e\": rpc error: code = NotFound desc = could not find container \"07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e\": container with ID starting with 07b3f70466150c8111a6d9c57e62add8224b01797742d9ac906ed529299cc12e not found: ID does not exist"
Sep 17 16:44:22.701327 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.701233    2566 scope.go:117] "RemoveContainer" containerID="b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579"
Sep 17 16:44:22.701489 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.701472    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579"} err="failed to get container status \"b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579\": rpc error: code = NotFound desc = could not find container \"b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579\": container with ID starting with b2f7cf2c930621f0b7b74272a298b9ce4de3ddd393f185f9df39a53652439579 not found: ID does not exist"
Sep 17 16:44:22.703581 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703563    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:44:22.703810 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703799    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="config-reloader"
Sep 17 16:44:22.703848 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703813    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="config-reloader"
Sep 17 16:44:22.703848 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703828    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy-metric"
Sep 17 16:44:22.703848 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703834    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy-metric"
Sep 17 16:44:22.703848 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703840    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy"
Sep 17 16:44:22.703848 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703846    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703853    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="init-config-reloader"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703858    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="init-config-reloader"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703864    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="alertmanager"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703869    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="alertmanager"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703875    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy-web"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703880    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy-web"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703889    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="prom-label-proxy"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703895    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="prom-label-proxy"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703939    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="config-reloader"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703946    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy-web"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703953    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="prom-label-proxy"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703959    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy-metric"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703964    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="alertmanager"
Sep 17 16:44:22.703988 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.703970    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" containerName="kube-rbac-proxy"
Sep 17 16:44:22.707371 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.707351    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.710059 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.709873    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 16:44:22.710059 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.709962    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 16:44:22.711197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.710139    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 16:44:22.711197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.710142    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8k6kh\""
Sep 17 16:44:22.711197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.710222    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 16:44:22.711197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.710441    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 16:44:22.711197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.710483    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 16:44:22.711197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.710441    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 16:44:22.711197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.710672    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 16:44:22.715535 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.715517    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 16:44:22.722824 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.722799    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:44:22.810821 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.810753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-config-volume\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.810821 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.810790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18672a27-e8ea-4f46-a9bf-fc736c50a419-config-out\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.810821 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.810815    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.810984 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.810891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.810984 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.810927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.810984 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.810948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18672a27-e8ea-4f46-a9bf-fc736c50a419-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.810984 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.810971    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18672a27-e8ea-4f46-a9bf-fc736c50a419-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.811168 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.811008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18672a27-e8ea-4f46-a9bf-fc736c50a419-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.811168 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.811022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.811168 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.811035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18672a27-e8ea-4f46-a9bf-fc736c50a419-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.811168 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.811059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjkq\" (UniqueName: \"kubernetes.io/projected/18672a27-e8ea-4f46-a9bf-fc736c50a419-kube-api-access-lpjkq\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.811168 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.811074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.811168 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.811136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-web-config\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.911457 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.911433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-web-config\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.911547 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.911465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-config-volume\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.911547 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.911480    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18672a27-e8ea-4f46-a9bf-fc736c50a419-config-out\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.911770 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.911752    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.911817 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.911801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.911865 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.911840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.911916 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.911864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18672a27-e8ea-4f46-a9bf-fc736c50a419-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.911968 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.911930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18672a27-e8ea-4f46-a9bf-fc736c50a419-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.912055 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.911966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18672a27-e8ea-4f46-a9bf-fc736c50a419-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.912055 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.911990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.912055 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.912012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18672a27-e8ea-4f46-a9bf-fc736c50a419-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.913090 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.912221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjkq\" (UniqueName: \"kubernetes.io/projected/18672a27-e8ea-4f46-a9bf-fc736c50a419-kube-api-access-lpjkq\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.913090 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.912291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.913090 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.912626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18672a27-e8ea-4f46-a9bf-fc736c50a419-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.913090 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.912291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18672a27-e8ea-4f46-a9bf-fc736c50a419-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.913428 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.913100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18672a27-e8ea-4f46-a9bf-fc736c50a419-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.914407 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.914381    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-config-volume\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.914803 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.914772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.914891 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.914844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.915137 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.915112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-web-config\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.915486 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.915466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.915725 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.915709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18672a27-e8ea-4f46-a9bf-fc736c50a419-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.915770 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.915711    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.915809 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.915787    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18672a27-e8ea-4f46-a9bf-fc736c50a419-config-out\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.916230 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.916216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18672a27-e8ea-4f46-a9bf-fc736c50a419-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:22.920387 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:22.920364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjkq\" (UniqueName: \"kubernetes.io/projected/18672a27-e8ea-4f46-a9bf-fc736c50a419-kube-api-access-lpjkq\") pod \"alertmanager-main-0\" (UID: \"18672a27-e8ea-4f46-a9bf-fc736c50a419\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:23.017219 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:23.017196    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:44:23.296647 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:23.296617    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:44:23.298192 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:44:23.298168    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18672a27_e8ea_4f46_a9bf_fc736c50a419.slice/crio-de5d0eb2dc7250bf926fef5124016b2a3e41427a4d1d716823f38b12f71fe8db WatchSource:0}: Error finding container de5d0eb2dc7250bf926fef5124016b2a3e41427a4d1d716823f38b12f71fe8db: Status 404 returned error can't find the container with id de5d0eb2dc7250bf926fef5124016b2a3e41427a4d1d716823f38b12f71fe8db
Sep 17 16:44:23.655966 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:23.655890    2566 generic.go:356] "Generic (PLEG): container finished" podID="18672a27-e8ea-4f46-a9bf-fc736c50a419" containerID="3ba404f894a6a4a0d3be42bc11749c6a877e46262e5b6b214ce6153489dd278d" exitCode=0
Sep 17 16:44:23.656354 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:23.655974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18672a27-e8ea-4f46-a9bf-fc736c50a419","Type":"ContainerDied","Data":"3ba404f894a6a4a0d3be42bc11749c6a877e46262e5b6b214ce6153489dd278d"}
Sep 17 16:44:23.656354 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:23.656041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18672a27-e8ea-4f46-a9bf-fc736c50a419","Type":"ContainerStarted","Data":"de5d0eb2dc7250bf926fef5124016b2a3e41427a4d1d716823f38b12f71fe8db"}
Sep 17 16:44:23.809121 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:23.809096    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc855443-4820-4b92-8abc-0afaed1cf7da" path="/var/lib/kubelet/pods/dc855443-4820-4b92-8abc-0afaed1cf7da/volumes"
Sep 17 16:44:24.664921 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:24.664886    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18672a27-e8ea-4f46-a9bf-fc736c50a419","Type":"ContainerStarted","Data":"726d849dce6412bb7367ea73e0adeb7d564b1f84ca263c92bda2861ccf133798"}
Sep 17 16:44:24.665350 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:24.664977    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18672a27-e8ea-4f46-a9bf-fc736c50a419","Type":"ContainerStarted","Data":"088b0d44a6da5b86687f6ec03aca2b8409d84911f77b59d4c7e0e2d0e3251c7a"}
Sep 17 16:44:24.665350 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:24.665004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18672a27-e8ea-4f46-a9bf-fc736c50a419","Type":"ContainerStarted","Data":"58f8f00a6a3904bf0ea856b52df4504182a74566e02c263fda21783e5b8ffb52"}
Sep 17 16:44:24.665350 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:24.665031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18672a27-e8ea-4f46-a9bf-fc736c50a419","Type":"ContainerStarted","Data":"cc89462f28b1fe4a2d084afa2e492ffaf8cd1a45d7c3c65da0bb3ea0b80796aa"}
Sep 17 16:44:24.665350 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:24.665055    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18672a27-e8ea-4f46-a9bf-fc736c50a419","Type":"ContainerStarted","Data":"7d5ae596e552fc074bf2a29117f2bcc9a1ec2807b4dab7e446b33c3fba75b962"}
Sep 17 16:44:24.665350 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:24.665077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18672a27-e8ea-4f46-a9bf-fc736c50a419","Type":"ContainerStarted","Data":"4cb1e5e0f88da8d07b2f860cf90f8ce620ad54d0ba0ae6c3cedb4ff52c83a9d7"}
Sep 17 16:44:24.690954 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:24.690911    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.677004079 podStartE2EDuration="2.690896616s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.892537ms" isStatefulPod=false podCreationTimestamp="2026-09-17 16:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:24.689418632 +0000 UTC m=+173.401425899" watchObservedRunningTime="2026-09-17 16:44:24.690896616 +0000 UTC m=+173.402903885"
Sep 17 16:44:25.142403 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.142368    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8846c548-t74j2"]
Sep 17 16:44:25.145921 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.145894    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.148165 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.148141    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 16:44:25.148684 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.148664    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 16:44:25.148903 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.148874    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 16:44:25.149037 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.149013    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 16:44:25.149102 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.149092    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 16:44:25.149832 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.149815    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xxqmq\""
Sep 17 16:44:25.155085 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.155064    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 16:44:25.157528 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.157503    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8846c548-t74j2"]
Sep 17 16:44:25.231568 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.231542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-secret-telemeter-client\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.231678 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.231575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-serving-certs-ca-bundle\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.231678 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.231605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml556\" (UniqueName: \"kubernetes.io/projected/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-kube-api-access-ml556\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.231779 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.231674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-federate-client-tls\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.231779 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.231734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-metrics-client-ca\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.231779 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.231750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.231779 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.231770    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-telemeter-client-tls\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.231963 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.231820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.332789 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.332761    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-metrics-client-ca\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.332789 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.332791    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.333008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.332808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-telemeter-client-tls\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.333008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.332826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.333008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.332848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-secret-telemeter-client\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.333008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.332868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-serving-certs-ca-bundle\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.333008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.332923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml556\" (UniqueName: \"kubernetes.io/projected/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-kube-api-access-ml556\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.333236 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.333089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-federate-client-tls\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.333644 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.333619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-metrics-client-ca\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.333752 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.333619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-serving-certs-ca-bundle\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.334079 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.334055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.336016 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.335990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-federate-client-tls\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.336881 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.336443    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-telemeter-client-tls\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.336881 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.336823    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.337397 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.337376    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-secret-telemeter-client\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.343065 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.343035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml556\" (UniqueName: \"kubernetes.io/projected/eabba0a4-769e-47f9-a74a-55dbf9a1fe5c-kube-api-access-ml556\") pod \"telemeter-client-8846c548-t74j2\" (UID: \"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c\") " pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.457407 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.457325    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8846c548-t74j2"
Sep 17 16:44:25.746740 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:25.746698    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8846c548-t74j2"]
Sep 17 16:44:25.749951 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:44:25.749923    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabba0a4_769e_47f9_a74a_55dbf9a1fe5c.slice/crio-bd77b8096aa6c5d39fbe90ec0c8230b4dcfd2e0433c09380f0f907aec11f7598 WatchSource:0}: Error finding container bd77b8096aa6c5d39fbe90ec0c8230b4dcfd2e0433c09380f0f907aec11f7598: Status 404 returned error can't find the container with id bd77b8096aa6c5d39fbe90ec0c8230b4dcfd2e0433c09380f0f907aec11f7598
Sep 17 16:44:26.675225 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:26.675180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8846c548-t74j2" event={"ID":"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c","Type":"ContainerStarted","Data":"bd77b8096aa6c5d39fbe90ec0c8230b4dcfd2e0433c09380f0f907aec11f7598"}
Sep 17 16:44:27.681304 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:27.681209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8846c548-t74j2" event={"ID":"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c","Type":"ContainerStarted","Data":"c20abd26652e02ebbbb906505df0ee0fa390214789233a7d8385ceeaad4da024"}
Sep 17 16:44:27.681651 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:27.681315    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8846c548-t74j2" event={"ID":"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c","Type":"ContainerStarted","Data":"5b71486fea7eda0ae66620a4888acc2ba3f3d8ddd2cf0d2adcba558a7032cd56"}
Sep 17 16:44:27.681651 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:27.681343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8846c548-t74j2" event={"ID":"eabba0a4-769e-47f9-a74a-55dbf9a1fe5c","Type":"ContainerStarted","Data":"5639021dfa91054796f5c46ebd2c8bea10e416515b4544184f370d56d465240e"}
Sep 17 16:44:27.704742 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:27.704694    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8846c548-t74j2" podStartSLOduration=1.046942104 podStartE2EDuration="2.704678978s" totalImagesPullingTime="1.657736874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:27.703950845 +0000 UTC m=+176.415958114" watchObservedRunningTime="2026-09-17 16:44:27.704678978 +0000 UTC m=+176.416686245"
Sep 17 16:44:27.898735 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:27.898689    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:27.998965 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:27.998937    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:28.060835 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:28.060800    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:28.679248 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:28.679193    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:28.738947 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:28.738916    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:38.347753 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:38.347718    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9bd9f99f-f9grr"]
Sep 17 16:44:38.459409 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:38.459378    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:38.559469 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:38.559435    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:38.621409 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:44:38.621349    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:03.367941 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.367878    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c9bd9f99f-f9grr" podUID="a3bae121-5933-4f4e-870c-27be5f2c5c5d" containerName="console" containerID="cri-o://bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f" gracePeriod=15
Sep 17 16:45:03.601543 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.601519    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:45:03.712704 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.712632    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-config\") pod \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") "
Sep 17 16:45:03.712704 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.712675    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-serving-cert\") pod \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") "
Sep 17 16:45:03.712704 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.712701    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-service-ca\") pod \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") "
Sep 17 16:45:03.712953 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.712748    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-oauth-serving-cert\") pod \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") "
Sep 17 16:45:03.712953 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.712766    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-oauth-config\") pod \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") "
Sep 17 16:45:03.712953 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.712782    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74zmn\" (UniqueName: \"kubernetes.io/projected/a3bae121-5933-4f4e-870c-27be5f2c5c5d-kube-api-access-74zmn\") pod \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") "
Sep 17 16:45:03.712953 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.712820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-trusted-ca-bundle\") pod \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\" (UID: \"a3bae121-5933-4f4e-870c-27be5f2c5c5d\") "
Sep 17 16:45:03.713152 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.713101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-config" (OuterVolumeSpecName: "console-config") pod "a3bae121-5933-4f4e-870c-27be5f2c5c5d" (UID: "a3bae121-5933-4f4e-870c-27be5f2c5c5d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:03.713210 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.713154    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a3bae121-5933-4f4e-870c-27be5f2c5c5d" (UID: "a3bae121-5933-4f4e-870c-27be5f2c5c5d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:03.713289 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.713157    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-service-ca" (OuterVolumeSpecName: "service-ca") pod "a3bae121-5933-4f4e-870c-27be5f2c5c5d" (UID: "a3bae121-5933-4f4e-870c-27be5f2c5c5d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:03.713383 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.713364    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a3bae121-5933-4f4e-870c-27be5f2c5c5d" (UID: "a3bae121-5933-4f4e-870c-27be5f2c5c5d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:03.715103 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.715073    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3bae121-5933-4f4e-870c-27be5f2c5c5d-kube-api-access-74zmn" (OuterVolumeSpecName: "kube-api-access-74zmn") pod "a3bae121-5933-4f4e-870c-27be5f2c5c5d" (UID: "a3bae121-5933-4f4e-870c-27be5f2c5c5d"). InnerVolumeSpecName "kube-api-access-74zmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:45:03.715325 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.715301    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a3bae121-5933-4f4e-870c-27be5f2c5c5d" (UID: "a3bae121-5933-4f4e-870c-27be5f2c5c5d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:45:03.715419 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.715324    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a3bae121-5933-4f4e-870c-27be5f2c5c5d" (UID: "a3bae121-5933-4f4e-870c-27be5f2c5c5d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:45:03.813650 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.813628    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:45:03.813763 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.813651    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-serving-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:45:03.813763 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.813665    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-service-ca\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:45:03.813763 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.813676    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-oauth-serving-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:45:03.813763 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.813688    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3bae121-5933-4f4e-870c-27be5f2c5c5d-console-oauth-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:45:03.813763 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.813700    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74zmn\" (UniqueName: \"kubernetes.io/projected/a3bae121-5933-4f4e-870c-27be5f2c5c5d-kube-api-access-74zmn\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:45:03.813763 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.813713    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3bae121-5933-4f4e-870c-27be5f2c5c5d-trusted-ca-bundle\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:45:03.824965 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.824940    2566 generic.go:356] "Generic (PLEG): container finished" podID="a3bae121-5933-4f4e-870c-27be5f2c5c5d" containerID="bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f" exitCode=0
Sep 17 16:45:03.825045 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.824985    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd9f99f-f9grr"
Sep 17 16:45:03.825045 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.825013    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd9f99f-f9grr" event={"ID":"a3bae121-5933-4f4e-870c-27be5f2c5c5d","Type":"ContainerDied","Data":"bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f"}
Sep 17 16:45:03.825117 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.825057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd9f99f-f9grr" event={"ID":"a3bae121-5933-4f4e-870c-27be5f2c5c5d","Type":"ContainerDied","Data":"0e002177a5fb07def9db93dbecfa672efeeab4a9f5fba0382d1efa6da9320959"}
Sep 17 16:45:03.825117 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.825077    2566 scope.go:117] "RemoveContainer" containerID="bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f"
Sep 17 16:45:03.832998 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.832982    2566 scope.go:117] "RemoveContainer" containerID="bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f"
Sep 17 16:45:03.833298 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:45:03.833272    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f\": container with ID starting with bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f not found: ID does not exist" containerID="bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f"
Sep 17 16:45:03.833369 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.833307    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f"} err="failed to get container status \"bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f\": rpc error: code = NotFound desc = could not find container \"bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f\": container with ID starting with bcb957a7f18e166fc817764e88a20dffafa7d6492505a7a0a64ae117ba5beb1f not found: ID does not exist"
Sep 17 16:45:03.841693 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.841668    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9bd9f99f-f9grr"]
Sep 17 16:45:03.844806 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:03.844786    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c9bd9f99f-f9grr"]
Sep 17 16:45:05.809451 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:05.809414    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3bae121-5933-4f4e-870c-27be5f2c5c5d" path="/var/lib/kubelet/pods/a3bae121-5933-4f4e-870c-27be5f2c5c5d/volumes"
Sep 17 16:45:23.160377 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:23.160340    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:23.212854 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:23.212818    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:35.286382 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:35.286341    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:35.345387 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:35.345358    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:47.315602 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:47.315556    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:47.372265 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:47.372220    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:48.871538 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.871504    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ffbd6456b-sl8sf"]
Sep 17 16:45:48.871927 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.871795    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3bae121-5933-4f4e-870c-27be5f2c5c5d" containerName="console"
Sep 17 16:45:48.871927 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.871807    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3bae121-5933-4f4e-870c-27be5f2c5c5d" containerName="console"
Sep 17 16:45:48.871927 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.871874    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3bae121-5933-4f4e-870c-27be5f2c5c5d" containerName="console"
Sep 17 16:45:48.877320 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.877299    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:48.880715 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.880695    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 16:45:48.880869 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.880782    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mbrl8\""
Sep 17 16:45:48.881206 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.881189    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 16:45:48.881306 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.881233    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 16:45:48.883450 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.883433    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 16:45:48.884016 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.883714    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 16:45:48.884132 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.884115    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 16:45:48.884451 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.884429    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 16:45:48.889631 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.889611    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 16:45:48.892315 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.892296    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ffbd6456b-sl8sf"]
Sep 17 16:45:48.934475 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:48.934454    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:49.026975 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.026952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-service-ca\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.027065 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.026984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-oauth-serving-cert\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.027065 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.027003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ddws\" (UniqueName: \"kubernetes.io/projected/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-kube-api-access-4ddws\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.027065 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.027018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-trusted-ca-bundle\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.027161 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.027095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-serving-cert\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.027161 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.027115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-config\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.027161 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.027146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-oauth-config\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.127940 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.127882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-serving-cert\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.127940 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.127910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-config\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.127940 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.127936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-oauth-config\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.128131 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.127963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-service-ca\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.128131 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.127993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-oauth-serving-cert\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.128131 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.128015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ddws\" (UniqueName: \"kubernetes.io/projected/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-kube-api-access-4ddws\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.128131 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.128049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-trusted-ca-bundle\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.128782 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.128760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-config\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.128782 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.128774    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-oauth-serving-cert\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.128893 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.128850    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-service-ca\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.128893 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.128880    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-trusted-ca-bundle\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.130316 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.130292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-oauth-config\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.130508 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.130485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-serving-cert\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.135570 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.135547    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ddws\" (UniqueName: \"kubernetes.io/projected/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-kube-api-access-4ddws\") pod \"console-ffbd6456b-sl8sf\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") " pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.186393 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.186364    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:49.458863 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:49.458841    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ffbd6456b-sl8sf"]
Sep 17 16:45:49.461277 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:45:49.461229    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6854cfdb_2cda_47bd_b9c2_a9b10956d33d.slice/crio-54af1b821f55f5e8863c993a9092c32948321fbbd87494a0fd48aad13a20f0f8 WatchSource:0}: Error finding container 54af1b821f55f5e8863c993a9092c32948321fbbd87494a0fd48aad13a20f0f8: Status 404 returned error can't find the container with id 54af1b821f55f5e8863c993a9092c32948321fbbd87494a0fd48aad13a20f0f8
Sep 17 16:45:50.007333 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:50.007295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffbd6456b-sl8sf" event={"ID":"6854cfdb-2cda-47bd-b9c2-a9b10956d33d","Type":"ContainerStarted","Data":"e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9"}
Sep 17 16:45:50.007667 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:50.007345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffbd6456b-sl8sf" event={"ID":"6854cfdb-2cda-47bd-b9c2-a9b10956d33d","Type":"ContainerStarted","Data":"54af1b821f55f5e8863c993a9092c32948321fbbd87494a0fd48aad13a20f0f8"}
Sep 17 16:45:50.025192 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:50.025150    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ffbd6456b-sl8sf" podStartSLOduration=2.02513473 podStartE2EDuration="2.02513473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:45:50.02378653 +0000 UTC m=+258.735793811" watchObservedRunningTime="2026-09-17 16:45:50.02513473 +0000 UTC m=+258.737141997"
Sep 17 16:45:56.068543 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:56.068446    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:56.133422 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:56.133385    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:59.187280 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:59.187222    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:59.187694 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:59.187308    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:45:59.192178 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:45:59.192154    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:46:00.049464 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:46:00.049438    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:46:00.192469 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:46:00.192432    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:00.249629 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:46:00.249599    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:31.735759 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:46:31.735732    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 16:46:49.153822 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:46:49.153769    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:50.486959 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:46:50.486910    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:16.332470 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:16.332427    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:16.423026 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:16.422992    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:21.705129 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:21.705088    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:22.056782 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.056744    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:22.200033 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.199998    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:22.272844 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.272810    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"]
Sep 17 16:47:22.275300 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.275277    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:22.276013 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.275997    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:22.279058 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.279038    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-p8b27\""
Sep 17 16:47:22.279459 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.279441    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 16:47:22.280328 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.280305    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 16:47:22.280437 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.280327    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 16:47:22.280437 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.280373    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 16:47:22.280437 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.280404    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 16:47:22.291085 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.291057    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"]
Sep 17 16:47:22.379074 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.379014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/aa1d4c66-9b62-42d2-9b01-7db3f1e07465-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lxc49\" (UID: \"aa1d4c66-9b62-42d2-9b01-7db3f1e07465\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:22.379074 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.379049    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa1d4c66-9b62-42d2-9b01-7db3f1e07465-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lxc49\" (UID: \"aa1d4c66-9b62-42d2-9b01-7db3f1e07465\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:22.379216 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.379077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vckzq\" (UniqueName: \"kubernetes.io/projected/aa1d4c66-9b62-42d2-9b01-7db3f1e07465-kube-api-access-vckzq\") pod \"keda-metrics-apiserver-77857b7b98-lxc49\" (UID: \"aa1d4c66-9b62-42d2-9b01-7db3f1e07465\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:22.476788 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.476755    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:22.479728 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.479706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vckzq\" (UniqueName: \"kubernetes.io/projected/aa1d4c66-9b62-42d2-9b01-7db3f1e07465-kube-api-access-vckzq\") pod \"keda-metrics-apiserver-77857b7b98-lxc49\" (UID: \"aa1d4c66-9b62-42d2-9b01-7db3f1e07465\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:22.479821 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.479800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/aa1d4c66-9b62-42d2-9b01-7db3f1e07465-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lxc49\" (UID: \"aa1d4c66-9b62-42d2-9b01-7db3f1e07465\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:22.479882 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.479845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa1d4c66-9b62-42d2-9b01-7db3f1e07465-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lxc49\" (UID: \"aa1d4c66-9b62-42d2-9b01-7db3f1e07465\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:22.480178 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.480157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/aa1d4c66-9b62-42d2-9b01-7db3f1e07465-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lxc49\" (UID: \"aa1d4c66-9b62-42d2-9b01-7db3f1e07465\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:22.482267 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.482237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa1d4c66-9b62-42d2-9b01-7db3f1e07465-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lxc49\" (UID: \"aa1d4c66-9b62-42d2-9b01-7db3f1e07465\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:22.488598 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.488575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vckzq\" (UniqueName: \"kubernetes.io/projected/aa1d4c66-9b62-42d2-9b01-7db3f1e07465-kube-api-access-vckzq\") pod \"keda-metrics-apiserver-77857b7b98-lxc49\" (UID: \"aa1d4c66-9b62-42d2-9b01-7db3f1e07465\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:22.585775 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.585752    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:22.683173 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.683139    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:22.776357 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.776321    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:22.872002 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.871926    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"]
Sep 17 16:47:22.875128 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:47:22.875102    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1d4c66_9b62_42d2_9b01_7db3f1e07465.slice/crio-7fe3fef7660960b0f81a506162048f83cdea9e87340b1684de5e90cba223b6e6 WatchSource:0}: Error finding container 7fe3fef7660960b0f81a506162048f83cdea9e87340b1684de5e90cba223b6e6: Status 404 returned error can't find the container with id 7fe3fef7660960b0f81a506162048f83cdea9e87340b1684de5e90cba223b6e6
Sep 17 16:47:22.876498 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.876479    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:47:22.994111 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:22.994079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:23.249382 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:23.249281    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:23.377895 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:23.377856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49" event={"ID":"aa1d4c66-9b62-42d2-9b01-7db3f1e07465","Type":"ContainerStarted","Data":"7fe3fef7660960b0f81a506162048f83cdea9e87340b1684de5e90cba223b6e6"}
Sep 17 16:47:23.393014 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:23.392976    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:23.497093 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:23.497059    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:26.394211 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:26.394118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49" event={"ID":"aa1d4c66-9b62-42d2-9b01-7db3f1e07465","Type":"ContainerStarted","Data":"59bc79ffa98001d92d01a69434a0ce588e76b1d1db2de11e3c59fefc536bbce3"}
Sep 17 16:47:26.394643 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:26.394236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:26.415122 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:26.415078    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49" podStartSLOduration=1.244673052 podStartE2EDuration="4.41506397s" totalImagesPullingTime="3.170390918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:47:26.412647539 +0000 UTC m=+355.124654831" watchObservedRunningTime="2026-09-17 16:47:26.41506397 +0000 UTC m=+355.127071237"
Sep 17 16:47:28.368865 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:28.368828    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:28.433372 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:28.433332    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:37.403059 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:37.403030    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lxc49"
Sep 17 16:47:44.793941 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:44.793902    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:44.894716 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:47:44.894687    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:15.128331 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:15.128290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:15.183402 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:15.183369    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:41.371441 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.371397    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:41.425072 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.425045    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-v59ff"]
Sep 17 16:48:41.427623 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.427605    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:48:41.430076 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.430051    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 16:48:41.430076 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.430062    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:48:41.430242 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.430155    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:48:41.431145 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.431129    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-mttqp\""
Sep 17 16:48:41.437413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.437391    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-v59ff"]
Sep 17 16:48:41.452739 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.452711    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:41.523940 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.523914    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:41.569969 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.569943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrh8b\" (UniqueName: \"kubernetes.io/projected/bfa8798d-a482-4430-920a-057f68b479e0-kube-api-access-zrh8b\") pod \"kserve-controller-manager-846cf5d849-v59ff\" (UID: \"bfa8798d-a482-4430-920a-057f68b479e0\") " pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:48:41.570075 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.569991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa8798d-a482-4430-920a-057f68b479e0-cert\") pod \"kserve-controller-manager-846cf5d849-v59ff\" (UID: \"bfa8798d-a482-4430-920a-057f68b479e0\") " pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:48:41.670637 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.670582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrh8b\" (UniqueName: \"kubernetes.io/projected/bfa8798d-a482-4430-920a-057f68b479e0-kube-api-access-zrh8b\") pod \"kserve-controller-manager-846cf5d849-v59ff\" (UID: \"bfa8798d-a482-4430-920a-057f68b479e0\") " pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:48:41.670637 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.670630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa8798d-a482-4430-920a-057f68b479e0-cert\") pod \"kserve-controller-manager-846cf5d849-v59ff\" (UID: \"bfa8798d-a482-4430-920a-057f68b479e0\") " pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:48:41.670765 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:48:41.670727    2566 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 16:48:41.670816 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:48:41.670797    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa8798d-a482-4430-920a-057f68b479e0-cert podName:bfa8798d-a482-4430-920a-057f68b479e0 nodeName:}" failed. No retries permitted until 2026-09-17 16:48:42.170778632 +0000 UTC m=+430.882785877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bfa8798d-a482-4430-920a-057f68b479e0-cert") pod "kserve-controller-manager-846cf5d849-v59ff" (UID: "bfa8798d-a482-4430-920a-057f68b479e0") : secret "kserve-webhook-server-cert" not found
Sep 17 16:48:41.672900 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.672878    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:41.678676 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.678659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrh8b\" (UniqueName: \"kubernetes.io/projected/bfa8798d-a482-4430-920a-057f68b479e0-kube-api-access-zrh8b\") pod \"kserve-controller-manager-846cf5d849-v59ff\" (UID: \"bfa8798d-a482-4430-920a-057f68b479e0\") " pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:48:41.789613 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:41.789590    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:42.175086 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:42.175049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa8798d-a482-4430-920a-057f68b479e0-cert\") pod \"kserve-controller-manager-846cf5d849-v59ff\" (UID: \"bfa8798d-a482-4430-920a-057f68b479e0\") " pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:48:42.177641 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:42.177616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa8798d-a482-4430-920a-057f68b479e0-cert\") pod \"kserve-controller-manager-846cf5d849-v59ff\" (UID: \"bfa8798d-a482-4430-920a-057f68b479e0\") " pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:48:42.202408 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:42.202382    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:42.303196 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:42.303164    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:42.338291 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:42.338268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:48:42.403710 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:42.403683    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:42.607908 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:42.607880    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-v59ff"]
Sep 17 16:48:42.610638 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:48:42.610613    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa8798d_a482_4430_920a_057f68b479e0.slice/crio-96cebcf6da3e0a3e2e507ebc441fdc6fc542835fc1a7d8cdbf5990c1ad79eba6 WatchSource:0}: Error finding container 96cebcf6da3e0a3e2e507ebc441fdc6fc542835fc1a7d8cdbf5990c1ad79eba6: Status 404 returned error can't find the container with id 96cebcf6da3e0a3e2e507ebc441fdc6fc542835fc1a7d8cdbf5990c1ad79eba6
Sep 17 16:48:42.709344 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:42.709319    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-v59ff" event={"ID":"bfa8798d-a482-4430-920a-057f68b479e0","Type":"ContainerStarted","Data":"96cebcf6da3e0a3e2e507ebc441fdc6fc542835fc1a7d8cdbf5990c1ad79eba6"}
Sep 17 16:48:45.726964 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:45.726875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-v59ff" event={"ID":"bfa8798d-a482-4430-920a-057f68b479e0","Type":"ContainerStarted","Data":"37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2"}
Sep 17 16:48:45.727404 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:45.727014    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:48:45.743048 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:48:45.743001    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-846cf5d849-v59ff" podStartSLOduration=1.88241966 podStartE2EDuration="4.742986217s" totalImagesPullingTime="2.860566557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:48:45.742386609 +0000 UTC m=+434.454393877" watchObservedRunningTime="2026-09-17 16:48:45.742986217 +0000 UTC m=+434.454993485"
Sep 17 16:49:03.185610 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.185581    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-546bb6cdb4-bjfrg"]
Sep 17 16:49:03.192070 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.192047    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.198938 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.198913    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546bb6cdb4-bjfrg"]
Sep 17 16:49:03.229786 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.229753    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:03.325132 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.325104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-trusted-ca-bundle\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.325275 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.325145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a3584ac-7057-4dc8-a9ab-570b6833b830-console-oauth-config\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.325275 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.325173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-oauth-serving-cert\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.325275 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.325196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqr4b\" (UniqueName: \"kubernetes.io/projected/9a3584ac-7057-4dc8-a9ab-570b6833b830-kube-api-access-dqr4b\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.325275 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.325225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-service-ca\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.325440 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.325314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a3584ac-7057-4dc8-a9ab-570b6833b830-console-serving-cert\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.325440 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.325346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-console-config\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.426230 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.426192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-trusted-ca-bundle\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.426230 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.426236    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a3584ac-7057-4dc8-a9ab-570b6833b830-console-oauth-config\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.426486 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.426272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-oauth-serving-cert\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.426486 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.426288    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqr4b\" (UniqueName: \"kubernetes.io/projected/9a3584ac-7057-4dc8-a9ab-570b6833b830-kube-api-access-dqr4b\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.426486 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.426310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-service-ca\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.426486 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.426455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a3584ac-7057-4dc8-a9ab-570b6833b830-console-serving-cert\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.426687 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.426501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-console-config\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.427138 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.427108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-oauth-serving-cert\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.427295 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.427150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-service-ca\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.427295 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.427209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-console-config\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.427295 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.427289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a3584ac-7057-4dc8-a9ab-570b6833b830-trusted-ca-bundle\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.428887 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.428867    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a3584ac-7057-4dc8-a9ab-570b6833b830-console-serving-cert\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.428989 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.428891    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a3584ac-7057-4dc8-a9ab-570b6833b830-console-oauth-config\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.433792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.433770    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqr4b\" (UniqueName: \"kubernetes.io/projected/9a3584ac-7057-4dc8-a9ab-570b6833b830-kube-api-access-dqr4b\") pod \"console-546bb6cdb4-bjfrg\" (UID: \"9a3584ac-7057-4dc8-a9ab-570b6833b830\") " pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.502962 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.502932    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:03.775870 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.775803    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546bb6cdb4-bjfrg"]
Sep 17 16:49:03.778620 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:49:03.778581    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3584ac_7057_4dc8_a9ab_570b6833b830.slice/crio-55614485fd91c8d3e3f9498fc2a45b8edf236d0a94b40e5da030ab2d3ff39269 WatchSource:0}: Error finding container 55614485fd91c8d3e3f9498fc2a45b8edf236d0a94b40e5da030ab2d3ff39269: Status 404 returned error can't find the container with id 55614485fd91c8d3e3f9498fc2a45b8edf236d0a94b40e5da030ab2d3ff39269
Sep 17 16:49:03.808759 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:03.808734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546bb6cdb4-bjfrg" event={"ID":"9a3584ac-7057-4dc8-a9ab-570b6833b830","Type":"ContainerStarted","Data":"55614485fd91c8d3e3f9498fc2a45b8edf236d0a94b40e5da030ab2d3ff39269"}
Sep 17 16:49:04.814273 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:04.814217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546bb6cdb4-bjfrg" event={"ID":"9a3584ac-7057-4dc8-a9ab-570b6833b830","Type":"ContainerStarted","Data":"facb2d95ac91838b21881d17362b815e104f1197401a7fa727cc00311a50b437"}
Sep 17 16:49:04.832373 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:04.832324    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-546bb6cdb4-bjfrg" podStartSLOduration=1.8323089019999999 podStartE2EDuration="1.832308902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:49:04.830944657 +0000 UTC m=+453.542951926" watchObservedRunningTime="2026-09-17 16:49:04.832308902 +0000 UTC m=+453.544316170"
Sep 17 16:49:06.909216 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:06.909183    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:07.069002 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:07.068966    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:13.503826 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:13.503787    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:13.504288 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:13.503885    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:13.508410 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:13.508390    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:13.851845 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:13.851776    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-546bb6cdb4-bjfrg"
Sep 17 16:49:13.901111 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:13.901065    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ffbd6456b-sl8sf"]
Sep 17 16:49:14.057727 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:14.057687    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:14.158122 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:14.158089    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:14.210705 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:14.210669    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:16.737070 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:16.737036    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:49:24.243151 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.243115    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-v59ff"]
Sep 17 16:49:24.243630 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.243323    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-846cf5d849-v59ff" podUID="bfa8798d-a482-4430-920a-057f68b479e0" containerName="manager" containerID="cri-o://37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2" gracePeriod=10
Sep 17 16:49:24.264305 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.264279    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-8t4bs"]
Sep 17 16:49:24.267360 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.267344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-8t4bs"
Sep 17 16:49:24.277455 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.277432    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-8t4bs"]
Sep 17 16:49:24.281917 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.281896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f61d2d-0246-4dd1-8baa-b8eb1603845d-cert\") pod \"kserve-controller-manager-846cf5d849-8t4bs\" (UID: \"d6f61d2d-0246-4dd1-8baa-b8eb1603845d\") " pod="kserve/kserve-controller-manager-846cf5d849-8t4bs"
Sep 17 16:49:24.282000 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.281981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9q6f\" (UniqueName: \"kubernetes.io/projected/d6f61d2d-0246-4dd1-8baa-b8eb1603845d-kube-api-access-q9q6f\") pod \"kserve-controller-manager-846cf5d849-8t4bs\" (UID: \"d6f61d2d-0246-4dd1-8baa-b8eb1603845d\") " pod="kserve/kserve-controller-manager-846cf5d849-8t4bs"
Sep 17 16:49:24.383133 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.383095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f61d2d-0246-4dd1-8baa-b8eb1603845d-cert\") pod \"kserve-controller-manager-846cf5d849-8t4bs\" (UID: \"d6f61d2d-0246-4dd1-8baa-b8eb1603845d\") " pod="kserve/kserve-controller-manager-846cf5d849-8t4bs"
Sep 17 16:49:24.383277 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.383201    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9q6f\" (UniqueName: \"kubernetes.io/projected/d6f61d2d-0246-4dd1-8baa-b8eb1603845d-kube-api-access-q9q6f\") pod \"kserve-controller-manager-846cf5d849-8t4bs\" (UID: \"d6f61d2d-0246-4dd1-8baa-b8eb1603845d\") " pod="kserve/kserve-controller-manager-846cf5d849-8t4bs"
Sep 17 16:49:24.385390 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.385363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f61d2d-0246-4dd1-8baa-b8eb1603845d-cert\") pod \"kserve-controller-manager-846cf5d849-8t4bs\" (UID: \"d6f61d2d-0246-4dd1-8baa-b8eb1603845d\") " pod="kserve/kserve-controller-manager-846cf5d849-8t4bs"
Sep 17 16:49:24.391802 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.391781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9q6f\" (UniqueName: \"kubernetes.io/projected/d6f61d2d-0246-4dd1-8baa-b8eb1603845d-kube-api-access-q9q6f\") pod \"kserve-controller-manager-846cf5d849-8t4bs\" (UID: \"d6f61d2d-0246-4dd1-8baa-b8eb1603845d\") " pod="kserve/kserve-controller-manager-846cf5d849-8t4bs"
Sep 17 16:49:24.484240 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.484217    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:49:24.585504 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.585440    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrh8b\" (UniqueName: \"kubernetes.io/projected/bfa8798d-a482-4430-920a-057f68b479e0-kube-api-access-zrh8b\") pod \"bfa8798d-a482-4430-920a-057f68b479e0\" (UID: \"bfa8798d-a482-4430-920a-057f68b479e0\") "
Sep 17 16:49:24.585504 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.585486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa8798d-a482-4430-920a-057f68b479e0-cert\") pod \"bfa8798d-a482-4430-920a-057f68b479e0\" (UID: \"bfa8798d-a482-4430-920a-057f68b479e0\") "
Sep 17 16:49:24.587461 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.587433    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa8798d-a482-4430-920a-057f68b479e0-cert" (OuterVolumeSpecName: "cert") pod "bfa8798d-a482-4430-920a-057f68b479e0" (UID: "bfa8798d-a482-4430-920a-057f68b479e0"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:49:24.587461 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.587444    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa8798d-a482-4430-920a-057f68b479e0-kube-api-access-zrh8b" (OuterVolumeSpecName: "kube-api-access-zrh8b") pod "bfa8798d-a482-4430-920a-057f68b479e0" (UID: "bfa8798d-a482-4430-920a-057f68b479e0"). InnerVolumeSpecName "kube-api-access-zrh8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:49:24.618217 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.618196    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-8t4bs"
Sep 17 16:49:24.686168 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.686141    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrh8b\" (UniqueName: \"kubernetes.io/projected/bfa8798d-a482-4430-920a-057f68b479e0-kube-api-access-zrh8b\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:49:24.686168 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.686167    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa8798d-a482-4430-920a-057f68b479e0-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:49:24.890201 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.889993    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-8t4bs"]
Sep 17 16:49:24.892637 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:49:24.892613    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f61d2d_0246_4dd1_8baa_b8eb1603845d.slice/crio-8f9626602791c09d7cec82664ba906e0e0f4b4f984f34c4c81bcc7c9c2dd7f8b WatchSource:0}: Error finding container 8f9626602791c09d7cec82664ba906e0e0f4b4f984f34c4c81bcc7c9c2dd7f8b: Status 404 returned error can't find the container with id 8f9626602791c09d7cec82664ba906e0e0f4b4f984f34c4c81bcc7c9c2dd7f8b
Sep 17 16:49:24.894408 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.894382    2566 generic.go:356] "Generic (PLEG): container finished" podID="bfa8798d-a482-4430-920a-057f68b479e0" containerID="37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2" exitCode=0
Sep 17 16:49:24.894507 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.894422    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-v59ff" event={"ID":"bfa8798d-a482-4430-920a-057f68b479e0","Type":"ContainerDied","Data":"37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2"}
Sep 17 16:49:24.894507 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.894457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-v59ff" event={"ID":"bfa8798d-a482-4430-920a-057f68b479e0","Type":"ContainerDied","Data":"96cebcf6da3e0a3e2e507ebc441fdc6fc542835fc1a7d8cdbf5990c1ad79eba6"}
Sep 17 16:49:24.894507 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.894488    2566 scope.go:117] "RemoveContainer" containerID="37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2"
Sep 17 16:49:24.894507 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.894428    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-846cf5d849-v59ff"
Sep 17 16:49:24.902754 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.902736    2566 scope.go:117] "RemoveContainer" containerID="37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2"
Sep 17 16:49:24.903026 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:49:24.903001    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2\": container with ID starting with 37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2 not found: ID does not exist" containerID="37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2"
Sep 17 16:49:24.903083 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.903035    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2"} err="failed to get container status \"37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2\": rpc error: code = NotFound desc = could not find container \"37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2\": container with ID starting with 37b238fb9b06b7241feaa9f771e2293f2895d6d2f7ae9a76865ac6006c28efe2 not found: ID does not exist"
Sep 17 16:49:24.915202 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.915179    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-v59ff"]
Sep 17 16:49:24.917568 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:24.917536    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-846cf5d849-v59ff"]
Sep 17 16:49:25.809407 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:25.809366    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfa8798d-a482-4430-920a-057f68b479e0" path="/var/lib/kubelet/pods/bfa8798d-a482-4430-920a-057f68b479e0/volumes"
Sep 17 16:49:25.901166 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:25.901130    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-8t4bs" event={"ID":"d6f61d2d-0246-4dd1-8baa-b8eb1603845d","Type":"ContainerStarted","Data":"cc9898a2264b5dbc6e3d4a53804cc7c9469554e94103c290e56da687e69fc5e7"}
Sep 17 16:49:25.901362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:25.901179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-846cf5d849-8t4bs" event={"ID":"d6f61d2d-0246-4dd1-8baa-b8eb1603845d","Type":"ContainerStarted","Data":"8f9626602791c09d7cec82664ba906e0e0f4b4f984f34c4c81bcc7c9c2dd7f8b"}
Sep 17 16:49:25.901362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:25.901219    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-846cf5d849-8t4bs"
Sep 17 16:49:25.919980 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:25.919935    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-846cf5d849-8t4bs" podStartSLOduration=1.602536108 podStartE2EDuration="1.919921533s" totalImagesPullingTime="317.385425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:49:25.917924429 +0000 UTC m=+474.629931710" watchObservedRunningTime="2026-09-17 16:49:25.919921533 +0000 UTC m=+474.631928883"
Sep 17 16:49:38.925307 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:38.925234    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ffbd6456b-sl8sf" podUID="6854cfdb-2cda-47bd-b9c2-a9b10956d33d" containerName="console" containerID="cri-o://e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9" gracePeriod=15
Sep 17 16:49:39.163622 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.163591    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:49:39.193527 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.193476    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-oauth-config\") pod \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") "
Sep 17 16:49:39.193527 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.193507    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-config\") pod \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") "
Sep 17 16:49:39.193703 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.193532    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-oauth-serving-cert\") pod \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") "
Sep 17 16:49:39.193703 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.193554    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ddws\" (UniqueName: \"kubernetes.io/projected/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-kube-api-access-4ddws\") pod \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") "
Sep 17 16:49:39.193703 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.193625    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-service-ca\") pod \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") "
Sep 17 16:49:39.193703 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.193669    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-trusted-ca-bundle\") pod \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") "
Sep 17 16:49:39.193897 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.193707    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-serving-cert\") pod \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\" (UID: \"6854cfdb-2cda-47bd-b9c2-a9b10956d33d\") "
Sep 17 16:49:39.193958 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.193901    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6854cfdb-2cda-47bd-b9c2-a9b10956d33d" (UID: "6854cfdb-2cda-47bd-b9c2-a9b10956d33d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:49:39.193958 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.193924    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-config" (OuterVolumeSpecName: "console-config") pod "6854cfdb-2cda-47bd-b9c2-a9b10956d33d" (UID: "6854cfdb-2cda-47bd-b9c2-a9b10956d33d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:49:39.194200 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.194157    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-service-ca" (OuterVolumeSpecName: "service-ca") pod "6854cfdb-2cda-47bd-b9c2-a9b10956d33d" (UID: "6854cfdb-2cda-47bd-b9c2-a9b10956d33d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:49:39.194395 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.194373    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6854cfdb-2cda-47bd-b9c2-a9b10956d33d" (UID: "6854cfdb-2cda-47bd-b9c2-a9b10956d33d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:49:39.195673 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.195630    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6854cfdb-2cda-47bd-b9c2-a9b10956d33d" (UID: "6854cfdb-2cda-47bd-b9c2-a9b10956d33d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:49:39.195783 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.195761    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6854cfdb-2cda-47bd-b9c2-a9b10956d33d" (UID: "6854cfdb-2cda-47bd-b9c2-a9b10956d33d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:49:39.195864 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.195849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-kube-api-access-4ddws" (OuterVolumeSpecName: "kube-api-access-4ddws") pod "6854cfdb-2cda-47bd-b9c2-a9b10956d33d" (UID: "6854cfdb-2cda-47bd-b9c2-a9b10956d33d"). InnerVolumeSpecName "kube-api-access-4ddws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:49:39.295197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.295173    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-service-ca\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:49:39.295197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.295194    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-trusted-ca-bundle\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:49:39.295343 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.295205    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-serving-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:49:39.295343 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.295214    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-oauth-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:49:39.295343 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.295222    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-console-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:49:39.295343 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.295231    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-oauth-serving-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:49:39.295343 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.295238    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ddws\" (UniqueName: \"kubernetes.io/projected/6854cfdb-2cda-47bd-b9c2-a9b10956d33d-kube-api-access-4ddws\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:49:39.960178 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.960144    2566 generic.go:356] "Generic (PLEG): container finished" podID="6854cfdb-2cda-47bd-b9c2-a9b10956d33d" containerID="e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9" exitCode=0
Sep 17 16:49:39.960532 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.960197    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffbd6456b-sl8sf"
Sep 17 16:49:39.960532 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.960225    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffbd6456b-sl8sf" event={"ID":"6854cfdb-2cda-47bd-b9c2-a9b10956d33d","Type":"ContainerDied","Data":"e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9"}
Sep 17 16:49:39.960532 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.960294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffbd6456b-sl8sf" event={"ID":"6854cfdb-2cda-47bd-b9c2-a9b10956d33d","Type":"ContainerDied","Data":"54af1b821f55f5e8863c993a9092c32948321fbbd87494a0fd48aad13a20f0f8"}
Sep 17 16:49:39.960532 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.960315    2566 scope.go:117] "RemoveContainer" containerID="e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9"
Sep 17 16:49:39.968903 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.968887    2566 scope.go:117] "RemoveContainer" containerID="e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9"
Sep 17 16:49:39.969179 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:49:39.969158    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9\": container with ID starting with e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9 not found: ID does not exist" containerID="e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9"
Sep 17 16:49:39.969230 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.969185    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9"} err="failed to get container status \"e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9\": rpc error: code = NotFound desc = could not find container \"e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9\": container with ID starting with e2269d456deccf09433b1de93279a23777b551fe3239e95fc4a9e2be8da39cc9 not found: ID does not exist"
Sep 17 16:49:39.977306 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.977285    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ffbd6456b-sl8sf"]
Sep 17 16:49:39.985451 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:39.985432    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ffbd6456b-sl8sf"]
Sep 17 16:49:41.808995 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:41.808947    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6854cfdb-2cda-47bd-b9c2-a9b10956d33d" path="/var/lib/kubelet/pods/6854cfdb-2cda-47bd-b9c2-a9b10956d33d/volumes"
Sep 17 16:49:55.631315 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:55.631274    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:55.732156 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:55.732125    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:56.911150 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:56.911121    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-846cf5d849-8t4bs"
Sep 17 16:49:58.201312 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.201281    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:58.235584 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.235557    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-zlbmp"]
Sep 17 16:49:58.235870 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.235857    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa8798d-a482-4430-920a-057f68b479e0" containerName="manager"
Sep 17 16:49:58.235870 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.235870    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa8798d-a482-4430-920a-057f68b479e0" containerName="manager"
Sep 17 16:49:58.235965 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.235880    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6854cfdb-2cda-47bd-b9c2-a9b10956d33d" containerName="console"
Sep 17 16:49:58.235965 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.235885    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6854cfdb-2cda-47bd-b9c2-a9b10956d33d" containerName="console"
Sep 17 16:49:58.235965 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.235945    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa8798d-a482-4430-920a-057f68b479e0" containerName="manager"
Sep 17 16:49:58.235965 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.235954    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6854cfdb-2cda-47bd-b9c2-a9b10956d33d" containerName="console"
Sep 17 16:49:58.238720 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.238696    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-zlbmp"
Sep 17 16:49:58.240959 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.240939    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 16:49:58.241091 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.241030    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-wz4ql\""
Sep 17 16:49:58.248106 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.248085    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-zlbmp"]
Sep 17 16:49:58.251069 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.251045    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-974pg"]
Sep 17 16:49:58.254142 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.254124    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:58.256652 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.256634    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 16:49:58.256736 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.256658    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 16:49:58.256781 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.256741    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-stq4d\""
Sep 17 16:49:58.263334 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.263314    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-974pg"]
Sep 17 16:49:58.274915 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.274875    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:58.336088 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.336062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/414c5dd5-a07a-4609-a58a-c3fa34686fd2-tls-certs\") pod \"model-serving-api-7b954ffc95-zlbmp\" (UID: \"414c5dd5-a07a-4609-a58a-c3fa34686fd2\") " pod="kserve/model-serving-api-7b954ffc95-zlbmp"
Sep 17 16:49:58.336221 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.336125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-metrics-certs\") pod \"odh-model-controller-7b966d7984-974pg\" (UID: \"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e\") " pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:58.336221 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.336194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj68d\" (UniqueName: \"kubernetes.io/projected/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-kube-api-access-kj68d\") pod \"odh-model-controller-7b966d7984-974pg\" (UID: \"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e\") " pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:58.336347 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.336231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjsm\" (UniqueName: \"kubernetes.io/projected/414c5dd5-a07a-4609-a58a-c3fa34686fd2-kube-api-access-hcjsm\") pod \"model-serving-api-7b954ffc95-zlbmp\" (UID: \"414c5dd5-a07a-4609-a58a-c3fa34686fd2\") " pod="kserve/model-serving-api-7b954ffc95-zlbmp"
Sep 17 16:49:58.336347 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.336278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-cert\") pod \"odh-model-controller-7b966d7984-974pg\" (UID: \"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e\") " pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:58.340629 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.340601    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:58.437311 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.437284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-metrics-certs\") pod \"odh-model-controller-7b966d7984-974pg\" (UID: \"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e\") " pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:58.437414 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.437317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj68d\" (UniqueName: \"kubernetes.io/projected/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-kube-api-access-kj68d\") pod \"odh-model-controller-7b966d7984-974pg\" (UID: \"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e\") " pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:58.437414 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.437340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjsm\" (UniqueName: \"kubernetes.io/projected/414c5dd5-a07a-4609-a58a-c3fa34686fd2-kube-api-access-hcjsm\") pod \"model-serving-api-7b954ffc95-zlbmp\" (UID: \"414c5dd5-a07a-4609-a58a-c3fa34686fd2\") " pod="kserve/model-serving-api-7b954ffc95-zlbmp"
Sep 17 16:49:58.437414 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.437395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-cert\") pod \"odh-model-controller-7b966d7984-974pg\" (UID: \"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e\") " pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:58.437547 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.437442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/414c5dd5-a07a-4609-a58a-c3fa34686fd2-tls-certs\") pod \"model-serving-api-7b954ffc95-zlbmp\" (UID: \"414c5dd5-a07a-4609-a58a-c3fa34686fd2\") " pod="kserve/model-serving-api-7b954ffc95-zlbmp"
Sep 17 16:49:58.437547 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:49:58.437520    2566 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 16:49:58.437661 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:49:58.437518    2566 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 16:49:58.437661 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:49:58.437577    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/414c5dd5-a07a-4609-a58a-c3fa34686fd2-tls-certs podName:414c5dd5-a07a-4609-a58a-c3fa34686fd2 nodeName:}" failed. No retries permitted until 2026-09-17 16:49:58.937562447 +0000 UTC m=+507.649569693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/414c5dd5-a07a-4609-a58a-c3fa34686fd2-tls-certs") pod "model-serving-api-7b954ffc95-zlbmp" (UID: "414c5dd5-a07a-4609-a58a-c3fa34686fd2") : secret "model-serving-api-tls" not found
Sep 17 16:49:58.437661 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:49:58.437641    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-cert podName:0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e nodeName:}" failed. No retries permitted until 2026-09-17 16:49:58.937623695 +0000 UTC m=+507.649630943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-cert") pod "odh-model-controller-7b966d7984-974pg" (UID: "0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e") : secret "odh-model-controller-webhook-cert" not found
Sep 17 16:49:58.439684 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.439660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-metrics-certs\") pod \"odh-model-controller-7b966d7984-974pg\" (UID: \"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e\") " pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:58.445761 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.445735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj68d\" (UniqueName: \"kubernetes.io/projected/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-kube-api-access-kj68d\") pod \"odh-model-controller-7b966d7984-974pg\" (UID: \"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e\") " pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:58.445761 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.445743    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjsm\" (UniqueName: \"kubernetes.io/projected/414c5dd5-a07a-4609-a58a-c3fa34686fd2-kube-api-access-hcjsm\") pod \"model-serving-api-7b954ffc95-zlbmp\" (UID: \"414c5dd5-a07a-4609-a58a-c3fa34686fd2\") " pod="kserve/model-serving-api-7b954ffc95-zlbmp"
Sep 17 16:49:58.558117 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.558093    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:58.659996 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.659974    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:58.941018 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.940946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/414c5dd5-a07a-4609-a58a-c3fa34686fd2-tls-certs\") pod \"model-serving-api-7b954ffc95-zlbmp\" (UID: \"414c5dd5-a07a-4609-a58a-c3fa34686fd2\") " pod="kserve/model-serving-api-7b954ffc95-zlbmp"
Sep 17 16:49:58.941018 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.941011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-cert\") pod \"odh-model-controller-7b966d7984-974pg\" (UID: \"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e\") " pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:58.943398 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.943377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/414c5dd5-a07a-4609-a58a-c3fa34686fd2-tls-certs\") pod \"model-serving-api-7b954ffc95-zlbmp\" (UID: \"414c5dd5-a07a-4609-a58a-c3fa34686fd2\") " pod="kserve/model-serving-api-7b954ffc95-zlbmp"
Sep 17 16:49:58.943510 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:58.943422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e-cert\") pod \"odh-model-controller-7b966d7984-974pg\" (UID: \"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e\") " pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:59.149395 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:59.149361    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-zlbmp"
Sep 17 16:49:59.164372 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:59.164342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:49:59.227872 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:59.227846    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:59.364174 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:59.364146    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:59.421473 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:59.421453    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-zlbmp"]
Sep 17 16:49:59.423428 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:49:59.423405    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414c5dd5_a07a_4609_a58a_c3fa34686fd2.slice/crio-8629e228a47305246f606ceb4e089612cac55fc6ffc7b33d743762f9f6f6f51f WatchSource:0}: Error finding container 8629e228a47305246f606ceb4e089612cac55fc6ffc7b33d743762f9f6f6f51f: Status 404 returned error can't find the container with id 8629e228a47305246f606ceb4e089612cac55fc6ffc7b33d743762f9f6f6f51f
Sep 17 16:49:59.429978 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:59.429951    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:49:59.445138 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:49:59.445115    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-974pg"]
Sep 17 16:49:59.448041 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:49:59.447990    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fca3dd1_cd4f_4cee_a6e6_b39170c9c78e.slice/crio-40888c516d8b235d39194013ac9a9060ad34c58c6971eccdbf0813866edcbb3c WatchSource:0}: Error finding container 40888c516d8b235d39194013ac9a9060ad34c58c6971eccdbf0813866edcbb3c: Status 404 returned error can't find the container with id 40888c516d8b235d39194013ac9a9060ad34c58c6971eccdbf0813866edcbb3c
Sep 17 16:50:00.044431 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:00.044396    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-974pg" event={"ID":"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e","Type":"ContainerStarted","Data":"40888c516d8b235d39194013ac9a9060ad34c58c6971eccdbf0813866edcbb3c"}
Sep 17 16:50:00.046731 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:00.046702    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-zlbmp" event={"ID":"414c5dd5-a07a-4609-a58a-c3fa34686fd2","Type":"ContainerStarted","Data":"8629e228a47305246f606ceb4e089612cac55fc6ffc7b33d743762f9f6f6f51f"}
Sep 17 16:50:04.068286 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:04.068231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-974pg" event={"ID":"0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e","Type":"ContainerStarted","Data":"f5c8cf0d78c5db7e6db68b4882eca6591428edd31f2b59307eab3dead7ef7bc5"}
Sep 17 16:50:04.068710 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:04.068320    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:50:04.069827 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:04.069802    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-zlbmp" event={"ID":"414c5dd5-a07a-4609-a58a-c3fa34686fd2","Type":"ContainerStarted","Data":"ab16d73cf03b3c383dcdc5f6865bfe0c92b59a8ba299cd3444c638d42fc97aed"}
Sep 17 16:50:04.069960 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:04.069941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-zlbmp"
Sep 17 16:50:04.084775 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:04.084733    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-974pg" podStartSLOduration=2.460938231 podStartE2EDuration="6.084721604s" totalImagesPullingTime="3.623783373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:50:04.083795637 +0000 UTC m=+512.795802904" watchObservedRunningTime="2026-09-17 16:50:04.084721604 +0000 UTC m=+512.796728865"
Sep 17 16:50:04.100175 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:04.100136    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-zlbmp" podStartSLOduration=2.495761218 podStartE2EDuration="6.10012558s" totalImagesPullingTime="3.604364362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:50:04.09875956 +0000 UTC m=+512.810766829" watchObservedRunningTime="2026-09-17 16:50:04.10012558 +0000 UTC m=+512.812132848"
Sep 17 16:50:15.076178 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:15.076147    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-974pg"
Sep 17 16:50:15.078103 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:15.078084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-zlbmp"
Sep 17 16:50:15.916698 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:15.916659    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-65jpf"]
Sep 17 16:50:15.920241 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:15.920222    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-65jpf"
Sep 17 16:50:15.922600 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:15.922579    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 16:50:15.922855 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:15.922831    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 16:50:15.922984 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:15.922858    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-42vlc\""
Sep 17 16:50:15.925927 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:15.925906    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-65jpf"]
Sep 17 16:50:16.066437 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:16.066409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9deb91f9-9168-4b0e-9e7a-4a18760d377b-model-cache\") pod \"s3-init-65jpf\" (UID: \"9deb91f9-9168-4b0e-9e7a-4a18760d377b\") " pod="kserve/s3-init-65jpf"
Sep 17 16:50:16.066528 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:16.066448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzmj6\" (UniqueName: \"kubernetes.io/projected/9deb91f9-9168-4b0e-9e7a-4a18760d377b-kube-api-access-nzmj6\") pod \"s3-init-65jpf\" (UID: \"9deb91f9-9168-4b0e-9e7a-4a18760d377b\") " pod="kserve/s3-init-65jpf"
Sep 17 16:50:16.167083 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:16.167021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9deb91f9-9168-4b0e-9e7a-4a18760d377b-model-cache\") pod \"s3-init-65jpf\" (UID: \"9deb91f9-9168-4b0e-9e7a-4a18760d377b\") " pod="kserve/s3-init-65jpf"
Sep 17 16:50:16.167083 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:16.167058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzmj6\" (UniqueName: \"kubernetes.io/projected/9deb91f9-9168-4b0e-9e7a-4a18760d377b-kube-api-access-nzmj6\") pod \"s3-init-65jpf\" (UID: \"9deb91f9-9168-4b0e-9e7a-4a18760d377b\") " pod="kserve/s3-init-65jpf"
Sep 17 16:50:16.167546 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:16.167507    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9deb91f9-9168-4b0e-9e7a-4a18760d377b-model-cache\") pod \"s3-init-65jpf\" (UID: \"9deb91f9-9168-4b0e-9e7a-4a18760d377b\") " pod="kserve/s3-init-65jpf"
Sep 17 16:50:16.175204 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:16.175183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzmj6\" (UniqueName: \"kubernetes.io/projected/9deb91f9-9168-4b0e-9e7a-4a18760d377b-kube-api-access-nzmj6\") pod \"s3-init-65jpf\" (UID: \"9deb91f9-9168-4b0e-9e7a-4a18760d377b\") " pod="kserve/s3-init-65jpf"
Sep 17 16:50:16.241301 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:16.241277    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-65jpf"
Sep 17 16:50:16.508884 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:16.508842    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-65jpf"]
Sep 17 16:50:16.511092 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:50:16.511054    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9deb91f9_9168_4b0e_9e7a_4a18760d377b.slice/crio-070e10833361f9e52f74a52959c85788e60da88673b8d2222a3fa371f2a6e76d WatchSource:0}: Error finding container 070e10833361f9e52f74a52959c85788e60da88673b8d2222a3fa371f2a6e76d: Status 404 returned error can't find the container with id 070e10833361f9e52f74a52959c85788e60da88673b8d2222a3fa371f2a6e76d
Sep 17 16:50:17.128126 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:17.128087    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-65jpf" event={"ID":"9deb91f9-9168-4b0e-9e7a-4a18760d377b","Type":"ContainerStarted","Data":"070e10833361f9e52f74a52959c85788e60da88673b8d2222a3fa371f2a6e76d"}
Sep 17 16:50:21.149097 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:21.149060    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-65jpf" event={"ID":"9deb91f9-9168-4b0e-9e7a-4a18760d377b","Type":"ContainerStarted","Data":"9130adb68b661d53cc9363f4a167af0f917640cc59c014dee9a71945a79f28df"}
Sep 17 16:50:23.159730 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:23.159694    2566 generic.go:356] "Generic (PLEG): container finished" podID="9deb91f9-9168-4b0e-9e7a-4a18760d377b" containerID="9130adb68b661d53cc9363f4a167af0f917640cc59c014dee9a71945a79f28df" exitCode=0
Sep 17 16:50:23.160143 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:23.159770    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-65jpf" event={"ID":"9deb91f9-9168-4b0e-9e7a-4a18760d377b","Type":"ContainerDied","Data":"9130adb68b661d53cc9363f4a167af0f917640cc59c014dee9a71945a79f28df"}
Sep 17 16:50:28.187093 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:28.187053    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-65jpf" event={"ID":"9deb91f9-9168-4b0e-9e7a-4a18760d377b","Type":"ContainerStarted","Data":"e88b0d7bbd264540f0ea2b2931ea939261194d10e6bbf92059ec0777469012f6"}
Sep 17 16:50:28.202308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:28.202247    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-65jpf" podStartSLOduration=2.866254412 podStartE2EDuration="13.202234134s" totalImagesPullingTime="8.348919617s" totalInitContainerRuntime="1.987060105s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:50:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:50:28.200976987 +0000 UTC m=+536.912984256" watchObservedRunningTime="2026-09-17 16:50:28.202234134 +0000 UTC m=+536.914241402"
Sep 17 16:50:47.960816 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:50:47.960763    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:33.484081 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:33.484044    2566 generic.go:356] "Generic (PLEG): container finished" podID="9deb91f9-9168-4b0e-9e7a-4a18760d377b" containerID="e88b0d7bbd264540f0ea2b2931ea939261194d10e6bbf92059ec0777469012f6" exitCode=0
Sep 17 16:51:33.484568 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:33.484117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-65jpf" event={"ID":"9deb91f9-9168-4b0e-9e7a-4a18760d377b","Type":"ContainerDied","Data":"e88b0d7bbd264540f0ea2b2931ea939261194d10e6bbf92059ec0777469012f6"}
Sep 17 16:51:34.611296 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:34.611267    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-65jpf"
Sep 17 16:51:34.683390 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:34.683360    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzmj6\" (UniqueName: \"kubernetes.io/projected/9deb91f9-9168-4b0e-9e7a-4a18760d377b-kube-api-access-nzmj6\") pod \"9deb91f9-9168-4b0e-9e7a-4a18760d377b\" (UID: \"9deb91f9-9168-4b0e-9e7a-4a18760d377b\") "
Sep 17 16:51:34.683512 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:34.683399    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9deb91f9-9168-4b0e-9e7a-4a18760d377b-model-cache\") pod \"9deb91f9-9168-4b0e-9e7a-4a18760d377b\" (UID: \"9deb91f9-9168-4b0e-9e7a-4a18760d377b\") "
Sep 17 16:51:34.685400 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:34.685379    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deb91f9-9168-4b0e-9e7a-4a18760d377b-kube-api-access-nzmj6" (OuterVolumeSpecName: "kube-api-access-nzmj6") pod "9deb91f9-9168-4b0e-9e7a-4a18760d377b" (UID: "9deb91f9-9168-4b0e-9e7a-4a18760d377b"). InnerVolumeSpecName "kube-api-access-nzmj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:51:34.705539 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:34.705515    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deb91f9-9168-4b0e-9e7a-4a18760d377b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9deb91f9-9168-4b0e-9e7a-4a18760d377b" (UID: "9deb91f9-9168-4b0e-9e7a-4a18760d377b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:51:34.784507 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:34.784453    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzmj6\" (UniqueName: \"kubernetes.io/projected/9deb91f9-9168-4b0e-9e7a-4a18760d377b-kube-api-access-nzmj6\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:51:34.784507 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:34.784475    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9deb91f9-9168-4b0e-9e7a-4a18760d377b-model-cache\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:51:35.493857 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:35.493834    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-65jpf"
Sep 17 16:51:35.493994 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:35.493858    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-65jpf" event={"ID":"9deb91f9-9168-4b0e-9e7a-4a18760d377b","Type":"ContainerDied","Data":"070e10833361f9e52f74a52959c85788e60da88673b8d2222a3fa371f2a6e76d"}
Sep 17 16:51:35.493994 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:35.493891    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="070e10833361f9e52f74a52959c85788e60da88673b8d2222a3fa371f2a6e76d"
Sep 17 16:51:36.191749 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.191718    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"]
Sep 17 16:51:36.192105 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.192043    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9deb91f9-9168-4b0e-9e7a-4a18760d377b" containerName="s3-init"
Sep 17 16:51:36.192105 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.192054    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9deb91f9-9168-4b0e-9e7a-4a18760d377b" containerName="s3-init"
Sep 17 16:51:36.192105 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.192065    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9deb91f9-9168-4b0e-9e7a-4a18760d377b" containerName="download-hf-model"
Sep 17 16:51:36.192105 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.192071    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9deb91f9-9168-4b0e-9e7a-4a18760d377b" containerName="download-hf-model"
Sep 17 16:51:36.192236 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.192143    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9deb91f9-9168-4b0e-9e7a-4a18760d377b" containerName="s3-init"
Sep 17 16:51:36.195157 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.195137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"
Sep 17 16:51:36.197664 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.197642    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 16:51:36.197755 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.197739    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-42vlc\""
Sep 17 16:51:36.205173 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.205078    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:36.205738 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.205720    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"]
Sep 17 16:51:36.296434 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.296408    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d21ff2a-bb54-4b37-914a-095985768c96-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-fbvsw\" (UID: \"4d21ff2a-bb54-4b37-914a-095985768c96\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"
Sep 17 16:51:36.296523 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.296438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpj97\" (UniqueName: \"kubernetes.io/projected/4d21ff2a-bb54-4b37-914a-095985768c96-kube-api-access-lpj97\") pod \"seaweedfs-tls-custom-6644bf5d6c-fbvsw\" (UID: \"4d21ff2a-bb54-4b37-914a-095985768c96\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"
Sep 17 16:51:36.397793 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.397772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d21ff2a-bb54-4b37-914a-095985768c96-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-fbvsw\" (UID: \"4d21ff2a-bb54-4b37-914a-095985768c96\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"
Sep 17 16:51:36.397877 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.397800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpj97\" (UniqueName: \"kubernetes.io/projected/4d21ff2a-bb54-4b37-914a-095985768c96-kube-api-access-lpj97\") pod \"seaweedfs-tls-custom-6644bf5d6c-fbvsw\" (UID: \"4d21ff2a-bb54-4b37-914a-095985768c96\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"
Sep 17 16:51:36.398105 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.398087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d21ff2a-bb54-4b37-914a-095985768c96-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-fbvsw\" (UID: \"4d21ff2a-bb54-4b37-914a-095985768c96\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"
Sep 17 16:51:36.406231 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.406206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpj97\" (UniqueName: \"kubernetes.io/projected/4d21ff2a-bb54-4b37-914a-095985768c96-kube-api-access-lpj97\") pod \"seaweedfs-tls-custom-6644bf5d6c-fbvsw\" (UID: \"4d21ff2a-bb54-4b37-914a-095985768c96\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"
Sep 17 16:51:36.503973 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.503950    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"
Sep 17 16:51:36.772400 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:36.772326    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"]
Sep 17 16:51:36.775241 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:51:36.775210    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d21ff2a_bb54_4b37_914a_095985768c96.slice/crio-3450ebb5bf38dd7757614748051692617cd385b634b56f8033eca305a5b9e90e WatchSource:0}: Error finding container 3450ebb5bf38dd7757614748051692617cd385b634b56f8033eca305a5b9e90e: Status 404 returned error can't find the container with id 3450ebb5bf38dd7757614748051692617cd385b634b56f8033eca305a5b9e90e
Sep 17 16:51:37.505656 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:37.505621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw" event={"ID":"4d21ff2a-bb54-4b37-914a-095985768c96","Type":"ContainerStarted","Data":"3450ebb5bf38dd7757614748051692617cd385b634b56f8033eca305a5b9e90e"}
Sep 17 16:51:40.521758 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:40.521724    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw" event={"ID":"4d21ff2a-bb54-4b37-914a-095985768c96","Type":"ContainerStarted","Data":"34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0"}
Sep 17 16:51:40.536308 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:40.536239    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw" podStartSLOduration=1.851233353 podStartE2EDuration="4.53622332s" totalImagesPullingTime="2.684989967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:51:40.535389944 +0000 UTC m=+609.247397212" watchObservedRunningTime="2026-09-17 16:51:40.53622332 +0000 UTC m=+609.248230587"
Sep 17 16:51:41.359952 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:41.359920    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"]
Sep 17 16:51:42.531405 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:42.531364    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw" podUID="4d21ff2a-bb54-4b37-914a-095985768c96" containerName="seaweedfs-tls-custom" containerID="cri-o://34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0" gracePeriod=30
Sep 17 16:51:43.769035 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:43.769010    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"
Sep 17 16:51:43.852727 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:43.852658    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d21ff2a-bb54-4b37-914a-095985768c96-data\") pod \"4d21ff2a-bb54-4b37-914a-095985768c96\" (UID: \"4d21ff2a-bb54-4b37-914a-095985768c96\") "
Sep 17 16:51:43.852858 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:43.852752    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpj97\" (UniqueName: \"kubernetes.io/projected/4d21ff2a-bb54-4b37-914a-095985768c96-kube-api-access-lpj97\") pod \"4d21ff2a-bb54-4b37-914a-095985768c96\" (UID: \"4d21ff2a-bb54-4b37-914a-095985768c96\") "
Sep 17 16:51:43.853996 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:43.853973    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d21ff2a-bb54-4b37-914a-095985768c96-data" (OuterVolumeSpecName: "data") pod "4d21ff2a-bb54-4b37-914a-095985768c96" (UID: "4d21ff2a-bb54-4b37-914a-095985768c96"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:51:43.854608 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:43.854589    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d21ff2a-bb54-4b37-914a-095985768c96-kube-api-access-lpj97" (OuterVolumeSpecName: "kube-api-access-lpj97") pod "4d21ff2a-bb54-4b37-914a-095985768c96" (UID: "4d21ff2a-bb54-4b37-914a-095985768c96"). InnerVolumeSpecName "kube-api-access-lpj97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:51:43.953846 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:43.953813    2566 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d21ff2a-bb54-4b37-914a-095985768c96-data\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:51:43.953846 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:43.953841    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpj97\" (UniqueName: \"kubernetes.io/projected/4d21ff2a-bb54-4b37-914a-095985768c96-kube-api-access-lpj97\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:51:44.541754 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.541715    2566 generic.go:356] "Generic (PLEG): container finished" podID="4d21ff2a-bb54-4b37-914a-095985768c96" containerID="34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0" exitCode=0
Sep 17 16:51:44.541966 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.541763    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"
Sep 17 16:51:44.541966 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.541813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw" event={"ID":"4d21ff2a-bb54-4b37-914a-095985768c96","Type":"ContainerDied","Data":"34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0"}
Sep 17 16:51:44.541966 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.541864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw" event={"ID":"4d21ff2a-bb54-4b37-914a-095985768c96","Type":"ContainerDied","Data":"3450ebb5bf38dd7757614748051692617cd385b634b56f8033eca305a5b9e90e"}
Sep 17 16:51:44.541966 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.541888    2566 scope.go:117] "RemoveContainer" containerID="34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0"
Sep 17 16:51:44.551605 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.551588    2566 scope.go:117] "RemoveContainer" containerID="34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0"
Sep 17 16:51:44.551898 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:51:44.551880    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0\": container with ID starting with 34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0 not found: ID does not exist" containerID="34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0"
Sep 17 16:51:44.551946 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.551908    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0"} err="failed to get container status \"34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0\": rpc error: code = NotFound desc = could not find container \"34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0\": container with ID starting with 34468657ddf9a4def41ca274d8dfebc4ab0c25b1953e5fb7ae60be7eaa4213d0 not found: ID does not exist"
Sep 17 16:51:44.562863 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.562837    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"]
Sep 17 16:51:44.566818 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.566797    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-fbvsw"]
Sep 17 16:51:44.590436 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.590412    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"]
Sep 17 16:51:44.590729 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.590717    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d21ff2a-bb54-4b37-914a-095985768c96" containerName="seaweedfs-tls-custom"
Sep 17 16:51:44.590781 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.590730    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d21ff2a-bb54-4b37-914a-095985768c96" containerName="seaweedfs-tls-custom"
Sep 17 16:51:44.590815 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.590805    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d21ff2a-bb54-4b37-914a-095985768c96" containerName="seaweedfs-tls-custom"
Sep 17 16:51:44.595056 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.595041    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"
Sep 17 16:51:44.597472 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.597452    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 17 16:51:44.597545 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.597494    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 16:51:44.597591 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.597541    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-42vlc\""
Sep 17 16:51:44.600792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.600770    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"]
Sep 17 16:51:44.758958 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.758928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/0ee6c2c0-9d1e-4ef9-b93b-a562d4201359-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-lsrbh\" (UID: \"0ee6c2c0-9d1e-4ef9-b93b-a562d4201359\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"
Sep 17 16:51:44.759076 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.758962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c2c0-9d1e-4ef9-b93b-a562d4201359-data\") pod \"seaweedfs-tls-custom-8b44d767d-lsrbh\" (UID: \"0ee6c2c0-9d1e-4ef9-b93b-a562d4201359\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"
Sep 17 16:51:44.759076 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.758987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skfk2\" (UniqueName: \"kubernetes.io/projected/0ee6c2c0-9d1e-4ef9-b93b-a562d4201359-kube-api-access-skfk2\") pod \"seaweedfs-tls-custom-8b44d767d-lsrbh\" (UID: \"0ee6c2c0-9d1e-4ef9-b93b-a562d4201359\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"
Sep 17 16:51:44.860397 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.860335    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/0ee6c2c0-9d1e-4ef9-b93b-a562d4201359-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-lsrbh\" (UID: \"0ee6c2c0-9d1e-4ef9-b93b-a562d4201359\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"
Sep 17 16:51:44.860397 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.860369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c2c0-9d1e-4ef9-b93b-a562d4201359-data\") pod \"seaweedfs-tls-custom-8b44d767d-lsrbh\" (UID: \"0ee6c2c0-9d1e-4ef9-b93b-a562d4201359\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"
Sep 17 16:51:44.860397 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.860385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skfk2\" (UniqueName: \"kubernetes.io/projected/0ee6c2c0-9d1e-4ef9-b93b-a562d4201359-kube-api-access-skfk2\") pod \"seaweedfs-tls-custom-8b44d767d-lsrbh\" (UID: \"0ee6c2c0-9d1e-4ef9-b93b-a562d4201359\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"
Sep 17 16:51:44.860776 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.860719    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ee6c2c0-9d1e-4ef9-b93b-a562d4201359-data\") pod \"seaweedfs-tls-custom-8b44d767d-lsrbh\" (UID: \"0ee6c2c0-9d1e-4ef9-b93b-a562d4201359\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"
Sep 17 16:51:44.862811 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.862792    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/0ee6c2c0-9d1e-4ef9-b93b-a562d4201359-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-lsrbh\" (UID: \"0ee6c2c0-9d1e-4ef9-b93b-a562d4201359\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"
Sep 17 16:51:44.868482 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.868461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skfk2\" (UniqueName: \"kubernetes.io/projected/0ee6c2c0-9d1e-4ef9-b93b-a562d4201359-kube-api-access-skfk2\") pod \"seaweedfs-tls-custom-8b44d767d-lsrbh\" (UID: \"0ee6c2c0-9d1e-4ef9-b93b-a562d4201359\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"
Sep 17 16:51:44.904742 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:44.904722    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"
Sep 17 16:51:45.174576 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:45.174553    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-lsrbh"]
Sep 17 16:51:45.177215 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:51:45.177187    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee6c2c0_9d1e_4ef9_b93b_a562d4201359.slice/crio-23b31beffb61fcda319514ee0ad5cb02b6eebbdbd93017c365140ea4e4098370 WatchSource:0}: Error finding container 23b31beffb61fcda319514ee0ad5cb02b6eebbdbd93017c365140ea4e4098370: Status 404 returned error can't find the container with id 23b31beffb61fcda319514ee0ad5cb02b6eebbdbd93017c365140ea4e4098370
Sep 17 16:51:45.548904 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:45.548869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh" event={"ID":"0ee6c2c0-9d1e-4ef9-b93b-a562d4201359","Type":"ContainerStarted","Data":"99d43dd35b729b8c7a6d1085f2eb31e6bb725614ca367ded5ba6b70e95c29fda"}
Sep 17 16:51:45.549035 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:45.548923    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh" event={"ID":"0ee6c2c0-9d1e-4ef9-b93b-a562d4201359","Type":"ContainerStarted","Data":"23b31beffb61fcda319514ee0ad5cb02b6eebbdbd93017c365140ea4e4098370"}
Sep 17 16:51:45.566640 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:45.566582    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-lsrbh" podStartSLOduration=1.299761095 podStartE2EDuration="1.566562912s" totalImagesPullingTime="266.801817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:51:45.56358663 +0000 UTC m=+614.275593897" watchObservedRunningTime="2026-09-17 16:51:45.566562912 +0000 UTC m=+614.278570181"
Sep 17 16:51:45.809659 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:45.809628    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d21ff2a-bb54-4b37-914a-095985768c96" path="/var/lib/kubelet/pods/4d21ff2a-bb54-4b37-914a-095985768c96/volumes"
Sep 17 16:51:45.877246 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:45.877223    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-cjbq7"]
Sep 17 16:51:45.880519 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:45.880504    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cjbq7"
Sep 17 16:51:45.882924 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:45.882907    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 16:51:45.887788 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:45.887740    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-cjbq7"]
Sep 17 16:51:45.971405 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:45.971374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11acc44c-aa4f-4098-8dc7-3d988d403efb-model-cache\") pod \"s3-tls-init-custom-cjbq7\" (UID: \"11acc44c-aa4f-4098-8dc7-3d988d403efb\") " pod="kserve/s3-tls-init-custom-cjbq7"
Sep 17 16:51:45.971547 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:45.971438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p452\" (UniqueName: \"kubernetes.io/projected/11acc44c-aa4f-4098-8dc7-3d988d403efb-kube-api-access-9p452\") pod \"s3-tls-init-custom-cjbq7\" (UID: \"11acc44c-aa4f-4098-8dc7-3d988d403efb\") " pod="kserve/s3-tls-init-custom-cjbq7"
Sep 17 16:51:46.072749 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:46.072685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p452\" (UniqueName: \"kubernetes.io/projected/11acc44c-aa4f-4098-8dc7-3d988d403efb-kube-api-access-9p452\") pod \"s3-tls-init-custom-cjbq7\" (UID: \"11acc44c-aa4f-4098-8dc7-3d988d403efb\") " pod="kserve/s3-tls-init-custom-cjbq7"
Sep 17 16:51:46.072749 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:46.072747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11acc44c-aa4f-4098-8dc7-3d988d403efb-model-cache\") pod \"s3-tls-init-custom-cjbq7\" (UID: \"11acc44c-aa4f-4098-8dc7-3d988d403efb\") " pod="kserve/s3-tls-init-custom-cjbq7"
Sep 17 16:51:46.073035 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:46.073020    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11acc44c-aa4f-4098-8dc7-3d988d403efb-model-cache\") pod \"s3-tls-init-custom-cjbq7\" (UID: \"11acc44c-aa4f-4098-8dc7-3d988d403efb\") " pod="kserve/s3-tls-init-custom-cjbq7"
Sep 17 16:51:46.081029 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:46.081007    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p452\" (UniqueName: \"kubernetes.io/projected/11acc44c-aa4f-4098-8dc7-3d988d403efb-kube-api-access-9p452\") pod \"s3-tls-init-custom-cjbq7\" (UID: \"11acc44c-aa4f-4098-8dc7-3d988d403efb\") " pod="kserve/s3-tls-init-custom-cjbq7"
Sep 17 16:51:46.195937 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:46.195907    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cjbq7"
Sep 17 16:51:46.463942 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:46.463917    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-cjbq7"]
Sep 17 16:51:46.465991 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:51:46.465965    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11acc44c_aa4f_4098_8dc7_3d988d403efb.slice/crio-b8a28e955960d5b9e5f0decca3e59c677d8e2252f0d62c280b6eb443a753d4e1 WatchSource:0}: Error finding container b8a28e955960d5b9e5f0decca3e59c677d8e2252f0d62c280b6eb443a753d4e1: Status 404 returned error can't find the container with id b8a28e955960d5b9e5f0decca3e59c677d8e2252f0d62c280b6eb443a753d4e1
Sep 17 16:51:46.554322 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:46.554293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cjbq7" event={"ID":"11acc44c-aa4f-4098-8dc7-3d988d403efb","Type":"ContainerStarted","Data":"721bddc3c70ee9b0ba2d2d59852317d6863770d8ac15802462ef305069a3571e"}
Sep 17 16:51:46.554433 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:46.554336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cjbq7" event={"ID":"11acc44c-aa4f-4098-8dc7-3d988d403efb","Type":"ContainerStarted","Data":"b8a28e955960d5b9e5f0decca3e59c677d8e2252f0d62c280b6eb443a753d4e1"}
Sep 17 16:51:48.565082 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:48.565001    2566 generic.go:356] "Generic (PLEG): container finished" podID="11acc44c-aa4f-4098-8dc7-3d988d403efb" containerID="721bddc3c70ee9b0ba2d2d59852317d6863770d8ac15802462ef305069a3571e" exitCode=0
Sep 17 16:51:48.565474 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:48.565076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cjbq7" event={"ID":"11acc44c-aa4f-4098-8dc7-3d988d403efb","Type":"ContainerDied","Data":"721bddc3c70ee9b0ba2d2d59852317d6863770d8ac15802462ef305069a3571e"}
Sep 17 16:51:49.571129 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:49.571095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cjbq7" event={"ID":"11acc44c-aa4f-4098-8dc7-3d988d403efb","Type":"ContainerStarted","Data":"99238556ebb2bfc9e5d6966f557ca835687bb1b20709513608c8f36c26ea7ab2"}
Sep 17 16:51:49.587894 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:49.587849    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-cjbq7" podStartSLOduration=2.859109153 podStartE2EDuration="4.587837023s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.72872787s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:51:49.586638616 +0000 UTC m=+618.298645885" watchObservedRunningTime="2026-09-17 16:51:49.587837023 +0000 UTC m=+618.299844290"
Sep 17 16:51:55.602117 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:55.602081    2566 generic.go:356] "Generic (PLEG): container finished" podID="11acc44c-aa4f-4098-8dc7-3d988d403efb" containerID="99238556ebb2bfc9e5d6966f557ca835687bb1b20709513608c8f36c26ea7ab2" exitCode=0
Sep 17 16:51:55.602497 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:55.602155    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cjbq7" event={"ID":"11acc44c-aa4f-4098-8dc7-3d988d403efb","Type":"ContainerDied","Data":"99238556ebb2bfc9e5d6966f557ca835687bb1b20709513608c8f36c26ea7ab2"}
Sep 17 16:51:56.742468 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:56.742446    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cjbq7"
Sep 17 16:51:56.758527 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:56.758504    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11acc44c-aa4f-4098-8dc7-3d988d403efb-model-cache\") pod \"11acc44c-aa4f-4098-8dc7-3d988d403efb\" (UID: \"11acc44c-aa4f-4098-8dc7-3d988d403efb\") "
Sep 17 16:51:56.758652 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:56.758599    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p452\" (UniqueName: \"kubernetes.io/projected/11acc44c-aa4f-4098-8dc7-3d988d403efb-kube-api-access-9p452\") pod \"11acc44c-aa4f-4098-8dc7-3d988d403efb\" (UID: \"11acc44c-aa4f-4098-8dc7-3d988d403efb\") "
Sep 17 16:51:56.760819 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:56.760782    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11acc44c-aa4f-4098-8dc7-3d988d403efb-kube-api-access-9p452" (OuterVolumeSpecName: "kube-api-access-9p452") pod "11acc44c-aa4f-4098-8dc7-3d988d403efb" (UID: "11acc44c-aa4f-4098-8dc7-3d988d403efb"). InnerVolumeSpecName "kube-api-access-9p452". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:51:56.791143 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:56.791119    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11acc44c-aa4f-4098-8dc7-3d988d403efb-model-cache" (OuterVolumeSpecName: "model-cache") pod "11acc44c-aa4f-4098-8dc7-3d988d403efb" (UID: "11acc44c-aa4f-4098-8dc7-3d988d403efb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:51:56.859286 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:56.859215    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p452\" (UniqueName: \"kubernetes.io/projected/11acc44c-aa4f-4098-8dc7-3d988d403efb-kube-api-access-9p452\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:51:56.859286 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:56.859235    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11acc44c-aa4f-4098-8dc7-3d988d403efb-model-cache\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:51:57.613904 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:57.613862    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cjbq7"
Sep 17 16:51:57.613904 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:57.613882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cjbq7" event={"ID":"11acc44c-aa4f-4098-8dc7-3d988d403efb","Type":"ContainerDied","Data":"b8a28e955960d5b9e5f0decca3e59c677d8e2252f0d62c280b6eb443a753d4e1"}
Sep 17 16:51:57.614107 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:57.613922    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8a28e955960d5b9e5f0decca3e59c677d8e2252f0d62c280b6eb443a753d4e1"
Sep 17 16:51:58.206262 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.206225    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"]
Sep 17 16:51:58.206599 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.206549    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11acc44c-aa4f-4098-8dc7-3d988d403efb" containerName="s3-tls-init-custom"
Sep 17 16:51:58.206599 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.206560    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="11acc44c-aa4f-4098-8dc7-3d988d403efb" containerName="s3-tls-init-custom"
Sep 17 16:51:58.206599 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.206581    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11acc44c-aa4f-4098-8dc7-3d988d403efb" containerName="download-hf-model"
Sep 17 16:51:58.206599 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.206587    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="11acc44c-aa4f-4098-8dc7-3d988d403efb" containerName="download-hf-model"
Sep 17 16:51:58.206727 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.206641    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="11acc44c-aa4f-4098-8dc7-3d988d403efb" containerName="s3-tls-init-custom"
Sep 17 16:51:58.211733 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.211717    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"
Sep 17 16:51:58.214302 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.214278    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 17 16:51:58.214421 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.214299    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 16:51:58.217504 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.217484    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"]
Sep 17 16:51:58.221888 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.221856    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:58.275007 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.274982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/34024e5e-282f-42cd-9f6e-24124ffe129c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-kdbhx\" (UID: \"34024e5e-282f-42cd-9f6e-24124ffe129c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"
Sep 17 16:51:58.275122 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.275037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mf4z\" (UniqueName: \"kubernetes.io/projected/34024e5e-282f-42cd-9f6e-24124ffe129c-kube-api-access-9mf4z\") pod \"seaweedfs-tls-serving-67b7477c6d-kdbhx\" (UID: \"34024e5e-282f-42cd-9f6e-24124ffe129c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"
Sep 17 16:51:58.275122 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.275056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/34024e5e-282f-42cd-9f6e-24124ffe129c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-kdbhx\" (UID: \"34024e5e-282f-42cd-9f6e-24124ffe129c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"
Sep 17 16:51:58.375961 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.375942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/34024e5e-282f-42cd-9f6e-24124ffe129c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-kdbhx\" (UID: \"34024e5e-282f-42cd-9f6e-24124ffe129c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"
Sep 17 16:51:58.376072 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.375996    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mf4z\" (UniqueName: \"kubernetes.io/projected/34024e5e-282f-42cd-9f6e-24124ffe129c-kube-api-access-9mf4z\") pod \"seaweedfs-tls-serving-67b7477c6d-kdbhx\" (UID: \"34024e5e-282f-42cd-9f6e-24124ffe129c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"
Sep 17 16:51:58.376072 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.376017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/34024e5e-282f-42cd-9f6e-24124ffe129c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-kdbhx\" (UID: \"34024e5e-282f-42cd-9f6e-24124ffe129c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"
Sep 17 16:51:58.376362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.376343    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/34024e5e-282f-42cd-9f6e-24124ffe129c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-kdbhx\" (UID: \"34024e5e-282f-42cd-9f6e-24124ffe129c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"
Sep 17 16:51:58.378291 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.378270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/34024e5e-282f-42cd-9f6e-24124ffe129c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-kdbhx\" (UID: \"34024e5e-282f-42cd-9f6e-24124ffe129c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"
Sep 17 16:51:58.384538 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.384515    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mf4z\" (UniqueName: \"kubernetes.io/projected/34024e5e-282f-42cd-9f6e-24124ffe129c-kube-api-access-9mf4z\") pod \"seaweedfs-tls-serving-67b7477c6d-kdbhx\" (UID: \"34024e5e-282f-42cd-9f6e-24124ffe129c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"
Sep 17 16:51:58.522207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.522184    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"
Sep 17 16:51:58.792285 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:58.792240    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx"]
Sep 17 16:51:58.794768 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:51:58.794733    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34024e5e_282f_42cd_9f6e_24124ffe129c.slice/crio-daad974216d3ea61831a0822e2d63055bc1880347f9e85b07b3bf5f9c6dc5717 WatchSource:0}: Error finding container daad974216d3ea61831a0822e2d63055bc1880347f9e85b07b3bf5f9c6dc5717: Status 404 returned error can't find the container with id daad974216d3ea61831a0822e2d63055bc1880347f9e85b07b3bf5f9c6dc5717
Sep 17 16:51:59.300096 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:59.300053    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:59.624314 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:59.624279    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx" event={"ID":"34024e5e-282f-42cd-9f6e-24124ffe129c","Type":"ContainerStarted","Data":"461c44333bc0c9a950bd3171c5df694563ceecdfe1b9d8f537cbd410c864f99f"}
Sep 17 16:51:59.624498 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:59.624333    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx" event={"ID":"34024e5e-282f-42cd-9f6e-24124ffe129c","Type":"ContainerStarted","Data":"daad974216d3ea61831a0822e2d63055bc1880347f9e85b07b3bf5f9c6dc5717"}
Sep 17 16:51:59.640371 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:51:59.640325    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-kdbhx" podStartSLOduration=1.382324697 podStartE2EDuration="1.640310624s" totalImagesPullingTime="257.985927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:51:59.639798877 +0000 UTC m=+628.351806146" watchObservedRunningTime="2026-09-17 16:51:59.640310624 +0000 UTC m=+628.352317893"
Sep 17 16:52:00.216737 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.216704    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-xbt6x"]
Sep 17 16:52:00.221366 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.221341    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-xbt6x"
Sep 17 16:52:00.226362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.226339    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 16:52:00.228322 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.228299    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-xbt6x"]
Sep 17 16:52:00.292732 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.292709    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtwz\" (UniqueName: \"kubernetes.io/projected/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-kube-api-access-9mtwz\") pod \"s3-tls-init-serving-xbt6x\" (UID: \"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06\") " pod="kserve/s3-tls-init-serving-xbt6x"
Sep 17 16:52:00.292853 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.292748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-model-cache\") pod \"s3-tls-init-serving-xbt6x\" (UID: \"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06\") " pod="kserve/s3-tls-init-serving-xbt6x"
Sep 17 16:52:00.393433 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.393410    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtwz\" (UniqueName: \"kubernetes.io/projected/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-kube-api-access-9mtwz\") pod \"s3-tls-init-serving-xbt6x\" (UID: \"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06\") " pod="kserve/s3-tls-init-serving-xbt6x"
Sep 17 16:52:00.393720 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.393450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-model-cache\") pod \"s3-tls-init-serving-xbt6x\" (UID: \"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06\") " pod="kserve/s3-tls-init-serving-xbt6x"
Sep 17 16:52:00.393802 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.393786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-model-cache\") pod \"s3-tls-init-serving-xbt6x\" (UID: \"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06\") " pod="kserve/s3-tls-init-serving-xbt6x"
Sep 17 16:52:00.402439 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.402411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtwz\" (UniqueName: \"kubernetes.io/projected/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-kube-api-access-9mtwz\") pod \"s3-tls-init-serving-xbt6x\" (UID: \"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06\") " pod="kserve/s3-tls-init-serving-xbt6x"
Sep 17 16:52:00.546026 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.546005    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-xbt6x"
Sep 17 16:52:00.815662 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:00.815639    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-xbt6x"]
Sep 17 16:52:00.818134 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:52:00.818095    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1f8e6b0_86ed_4a28_a53f_beeb06f7cd06.slice/crio-6461fe74f5a75a3bed63ce9470bc1f8b918237606d068815b19f9d4034c6e67d WatchSource:0}: Error finding container 6461fe74f5a75a3bed63ce9470bc1f8b918237606d068815b19f9d4034c6e67d: Status 404 returned error can't find the container with id 6461fe74f5a75a3bed63ce9470bc1f8b918237606d068815b19f9d4034c6e67d
Sep 17 16:52:01.636080 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:01.636046    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xbt6x" event={"ID":"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06","Type":"ContainerStarted","Data":"ce6c374b95e7b12de01b1be287ecaee415cc251f5d28ca80846beec1c23f1400"}
Sep 17 16:52:01.636649 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:01.636093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xbt6x" event={"ID":"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06","Type":"ContainerStarted","Data":"6461fe74f5a75a3bed63ce9470bc1f8b918237606d068815b19f9d4034c6e67d"}
Sep 17 16:52:03.647437 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:03.647401    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06" containerID="ce6c374b95e7b12de01b1be287ecaee415cc251f5d28ca80846beec1c23f1400" exitCode=0
Sep 17 16:52:03.647770 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:03.647475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xbt6x" event={"ID":"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06","Type":"ContainerDied","Data":"ce6c374b95e7b12de01b1be287ecaee415cc251f5d28ca80846beec1c23f1400"}
Sep 17 16:52:04.653484 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:04.653449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xbt6x" event={"ID":"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06","Type":"ContainerStarted","Data":"314f5b4f412301d3382e877aa23e5b9c13e340205af61ab40eec2495dd96dd61"}
Sep 17 16:52:04.669793 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:04.669747    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-xbt6x" podStartSLOduration=2.7014967260000002 podStartE2EDuration="4.669733323s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.968236597s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:04.668506624 +0000 UTC m=+633.380513893" watchObservedRunningTime="2026-09-17 16:52:04.669733323 +0000 UTC m=+633.381740590"
Sep 17 16:52:09.681135 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:09.681098    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06" containerID="314f5b4f412301d3382e877aa23e5b9c13e340205af61ab40eec2495dd96dd61" exitCode=0
Sep 17 16:52:09.681526 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:09.681165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xbt6x" event={"ID":"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06","Type":"ContainerDied","Data":"314f5b4f412301d3382e877aa23e5b9c13e340205af61ab40eec2495dd96dd61"}
Sep 17 16:52:10.822452 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:10.822423    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-xbt6x"
Sep 17 16:52:10.884008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:10.883981    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mtwz\" (UniqueName: \"kubernetes.io/projected/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-kube-api-access-9mtwz\") pod \"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06\" (UID: \"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06\") "
Sep 17 16:52:10.884136 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:10.884027    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-model-cache\") pod \"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06\" (UID: \"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06\") "
Sep 17 16:52:10.885995 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:10.885969    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-kube-api-access-9mtwz" (OuterVolumeSpecName: "kube-api-access-9mtwz") pod "d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06" (UID: "d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06"). InnerVolumeSpecName "kube-api-access-9mtwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:52:10.904788 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:10.904766    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06" (UID: "d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:10.985543 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:10.985524    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtwz\" (UniqueName: \"kubernetes.io/projected/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-kube-api-access-9mtwz\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:52:10.985634 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:10.985545    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06-model-cache\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:52:11.691644 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:11.691611    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xbt6x" event={"ID":"d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06","Type":"ContainerDied","Data":"6461fe74f5a75a3bed63ce9470bc1f8b918237606d068815b19f9d4034c6e67d"}
Sep 17 16:52:11.691644 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:11.691631    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-xbt6x"
Sep 17 16:52:11.691644 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:11.691649    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6461fe74f5a75a3bed63ce9470bc1f8b918237606d068815b19f9d4034c6e67d"
Sep 17 16:52:12.080266 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:12.080232    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:52:12.181369 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:12.181342    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:52:23.832496 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.832458    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:52:23.895092 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.895062    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"]
Sep 17 16:52:23.895424 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.895410    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06" containerName="download-hf-model"
Sep 17 16:52:23.895475 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.895424    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06" containerName="download-hf-model"
Sep 17 16:52:23.895475 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.895446    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06" containerName="s3-tls-init-serving"
Sep 17 16:52:23.895475 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.895452    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06" containerName="s3-tls-init-serving"
Sep 17 16:52:23.895593 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.895505    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06" containerName="s3-tls-init-serving"
Sep 17 16:52:23.900406 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.900387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:23.902778 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.902760    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:52:23.902920 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.902902    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 17 16:52:23.903078 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.903063    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:52:23.903170 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.903149    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:52:23.903313 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.903290    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:52:23.903912 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.903894    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 17 16:52:23.903998 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.903919    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 16:52:23.909646 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.909626    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"]
Sep 17 16:52:23.980446 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.980421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75gl\" (UniqueName: \"kubernetes.io/projected/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-api-access-q75gl\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:23.980568 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.980477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:23.980634 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.980566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:23.980634 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.980621    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0afb92f-53d7-4c5f-bc34-d6823baf898f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:23.980741 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.980703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:23.998947 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:23.998920    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:52:24.082182 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.082151    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:24.082372 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.082195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:24.082372 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.082218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0afb92f-53d7-4c5f-bc34-d6823baf898f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:24.082372 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.082240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:24.082372 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.082312    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75gl\" (UniqueName: \"kubernetes.io/projected/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-api-access-q75gl\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:24.082700 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.082640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:24.082871 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.082840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:24.083019 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.082924    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:24.084948 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.084921    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0afb92f-53d7-4c5f-bc34-d6823baf898f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:24.090019 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.089993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75gl\" (UniqueName: \"kubernetes.io/projected/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-api-access-q75gl\") pod \"isvc-sklearn-batcher-predictor-755dbf648-h8qwt\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:24.106855 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.106823    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:52:24.196918 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.196886    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:52:24.212386 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.212363    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:24.485813 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.485792    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"]
Sep 17 16:52:24.487853 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:52:24.487830    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0afb92f_53d7_4c5f_bc34_d6823baf898f.slice/crio-dd3fad1197d4183554af1f32d39fd591a26487fcad24e6b473bf8cd76227236b WatchSource:0}: Error finding container dd3fad1197d4183554af1f32d39fd591a26487fcad24e6b473bf8cd76227236b: Status 404 returned error can't find the container with id dd3fad1197d4183554af1f32d39fd591a26487fcad24e6b473bf8cd76227236b
Sep 17 16:52:24.755913 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.755825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" event={"ID":"f0afb92f-53d7-4c5f-bc34-d6823baf898f","Type":"ContainerStarted","Data":"1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa"}
Sep 17 16:52:24.755913 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:24.755898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" event={"ID":"f0afb92f-53d7-4c5f-bc34-d6823baf898f","Type":"ContainerStarted","Data":"dd3fad1197d4183554af1f32d39fd591a26487fcad24e6b473bf8cd76227236b"}
Sep 17 16:52:28.776105 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:28.776072    2566 generic.go:356] "Generic (PLEG): container finished" podID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerID="1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa" exitCode=0
Sep 17 16:52:28.776504 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:28.776120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" event={"ID":"f0afb92f-53d7-4c5f-bc34-d6823baf898f","Type":"ContainerDied","Data":"1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa"}
Sep 17 16:52:28.777346 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:28.777328    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:52:35.988413 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:35.988362    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:52:36.043034 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:36.042998    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:52:38.785810 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:38.785771    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:52:43.868438 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:43.868404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" event={"ID":"f0afb92f-53d7-4c5f-bc34-d6823baf898f","Type":"ContainerStarted","Data":"63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3"}
Sep 17 16:52:43.902341 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:43.902209    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:52:43.954331 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:43.954235    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"]
Sep 17 16:52:43.956920 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:43.956873    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:43.960011 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:43.959986    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 16:52:43.960149 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:43.960013    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 16:52:43.974589 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:43.974563    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"]
Sep 17 16:52:44.057820 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.057781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.057995 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.057833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.057995 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.057900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhzpd\" (UniqueName: \"kubernetes.io/projected/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-api-access-qhzpd\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.057995 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.057957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73696000-2f51-43d4-8c56-5a4c22fa81ce-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.058160 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.058002    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73696000-2f51-43d4-8c56-5a4c22fa81ce-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.158745 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.158707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.158919 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.158775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.158919 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.158816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhzpd\" (UniqueName: \"kubernetes.io/projected/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-api-access-qhzpd\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.158919 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.158856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73696000-2f51-43d4-8c56-5a4c22fa81ce-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.159086 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.158917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73696000-2f51-43d4-8c56-5a4c22fa81ce-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.159401 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.159365    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73696000-2f51-43d4-8c56-5a4c22fa81ce-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.159606 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.159552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.159761 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.159714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.161593 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.161572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73696000-2f51-43d4-8c56-5a4c22fa81ce-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.168207 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.168175    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhzpd\" (UniqueName: \"kubernetes.io/projected/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-api-access-qhzpd\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6h8gc\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.182650 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.182621    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:52:44.270601 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.270567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:52:44.563075 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.562995    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"]
Sep 17 16:52:44.567498 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:52:44.567467    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73696000_2f51_43d4_8c56_5a4c22fa81ce.slice/crio-427210ff58592ab92f1507c61ba2ff8fe6e1342c2a38cf773664e50f36913ae4 WatchSource:0}: Error finding container 427210ff58592ab92f1507c61ba2ff8fe6e1342c2a38cf773664e50f36913ae4: Status 404 returned error can't find the container with id 427210ff58592ab92f1507c61ba2ff8fe6e1342c2a38cf773664e50f36913ae4
Sep 17 16:52:44.876816 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.876735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" event={"ID":"73696000-2f51-43d4-8c56-5a4c22fa81ce","Type":"ContainerStarted","Data":"03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e"}
Sep 17 16:52:44.876816 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:44.876809    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" event={"ID":"73696000-2f51-43d4-8c56-5a4c22fa81ce","Type":"ContainerStarted","Data":"427210ff58592ab92f1507c61ba2ff8fe6e1342c2a38cf773664e50f36913ae4"}
Sep 17 16:52:46.890869 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:46.890830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" event={"ID":"f0afb92f-53d7-4c5f-bc34-d6823baf898f","Type":"ContainerStarted","Data":"81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5"}
Sep 17 16:52:48.903319 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:48.903226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" event={"ID":"f0afb92f-53d7-4c5f-bc34-d6823baf898f","Type":"ContainerStarted","Data":"c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6"}
Sep 17 16:52:48.903730 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:48.903374    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:48.904877 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:48.904850    2566 generic.go:356] "Generic (PLEG): container finished" podID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerID="03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e" exitCode=0
Sep 17 16:52:48.904990 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:48.904900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" event={"ID":"73696000-2f51-43d4-8c56-5a4c22fa81ce","Type":"ContainerDied","Data":"03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e"}
Sep 17 16:52:48.924757 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:48.924706    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podStartSLOduration=2.215300526 podStartE2EDuration="25.924689783s" totalImagesPullingTime="19.649902625s" totalInitContainerRuntime="4.059486632s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:52:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:48.924189886 +0000 UTC m=+677.636197166" watchObservedRunningTime="2026-09-17 16:52:48.924689783 +0000 UTC m=+677.636697054"
Sep 17 16:52:49.925093 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:49.920590    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:49.925093 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:49.920820    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:49.925093 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:49.924832    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:52:49.926134 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:49.925663    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:52:50.929311 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:50.928887    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:52:50.930192 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:50.930095    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:52:55.934040 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:55.934010    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:52:55.935055 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:55.935020    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:52:55.935769 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:52:55.935689    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:00.264502 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:00.264462    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:53:05.935271 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:05.935206    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:53:05.935833 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:05.935799    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:14.061500 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:14.061465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" event={"ID":"73696000-2f51-43d4-8c56-5a4c22fa81ce","Type":"ContainerStarted","Data":"81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9"}
Sep 17 16:53:14.061892 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:14.061550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" event={"ID":"73696000-2f51-43d4-8c56-5a4c22fa81ce","Type":"ContainerStarted","Data":"6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb"}
Sep 17 16:53:14.061892 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:14.061737    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:53:14.082054 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:14.082010    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podStartSLOduration=2.490157211 podStartE2EDuration="31.081995447s" totalImagesPullingTime="24.670289107s" totalInitContainerRuntime="3.921549129s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:53:14.079784962 +0000 UTC m=+702.791792241" watchObservedRunningTime="2026-09-17 16:53:14.081995447 +0000 UTC m=+702.794002715"
Sep 17 16:53:15.067080 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:15.067049    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:53:15.068326 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:15.068300    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:53:15.935059 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:15.935013    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:53:15.935450 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:15.935421    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:16.072393 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:16.072357    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:53:21.077474 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:21.077442    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:53:21.077945 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:21.077920    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:53:25.935126 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:25.935048    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:53:25.935513 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:25.935453    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:31.077985 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:31.077945    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:53:35.935536 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:35.935496    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:53:35.935999 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:35.935972    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:41.078593 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:41.078553    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:53:43.160723 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:43.160668    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:53:43.214111 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:43.214071    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:53:45.935122 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:45.935085    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:53:45.935588 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:45.935468    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:51.078348 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:51.078306    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:53:55.935461 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:55.935413    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:53:55.935881 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:53:55.935797    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:01.078091 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:01.078050    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:54:05.936374 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:05.936342    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:54:05.936857 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:05.936434    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:54:11.078637 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:11.078595    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:54:18.859315 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:18.859279    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"]
Sep 17 16:54:18.859693 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:18.859544    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" containerID="cri-o://63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3" gracePeriod=30
Sep 17 16:54:18.859693 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:18.859615    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" containerID="cri-o://c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6" gracePeriod=30
Sep 17 16:54:18.859693 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:18.859643    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kube-rbac-proxy" containerID="cri-o://81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5" gracePeriod=30
Sep 17 16:54:18.875074 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:18.875014    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:54:18.975382 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:18.975344    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:54:19.001837 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.001805    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"]
Sep 17 16:54:19.004584 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.004564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.006815 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.006796    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 17 16:54:19.006910 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.006856    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 17 16:54:19.016209 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.016187    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"]
Sep 17 16:54:19.159216 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.159133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.159216 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.159172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jwcn\" (UniqueName: \"kubernetes.io/projected/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-api-access-8jwcn\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.159479 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.159273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.159479 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.159329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25a12ba1-7ddb-4287-8607-e153d0e7e888-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.159479 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.159383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25a12ba1-7ddb-4287-8607-e153d0e7e888-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.260187 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.260151    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25a12ba1-7ddb-4287-8607-e153d0e7e888-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.260394 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.260200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25a12ba1-7ddb-4287-8607-e153d0e7e888-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.260394 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.260291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.260394 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.260318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jwcn\" (UniqueName: \"kubernetes.io/projected/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-api-access-8jwcn\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.260394 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.260372    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.260636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.260613    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25a12ba1-7ddb-4287-8607-e153d0e7e888-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.260953 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.260930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.261186 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.261165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.262787 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.262767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25a12ba1-7ddb-4287-8607-e153d0e7e888-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.268309 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.268286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jwcn\" (UniqueName: \"kubernetes.io/projected/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-api-access-8jwcn\") pod \"isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.289618 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.289588    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:54:19.315530 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.315507    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:19.380090 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.379900    2566 generic.go:356] "Generic (PLEG): container finished" podID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerID="81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5" exitCode=2
Sep 17 16:54:19.380090 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.380010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" event={"ID":"f0afb92f-53d7-4c5f-bc34-d6823baf898f","Type":"ContainerDied","Data":"81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5"}
Sep 17 16:54:19.590398 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:19.590295    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"]
Sep 17 16:54:19.592659 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:54:19.592627    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a12ba1_7ddb_4287_8607_e153d0e7e888.slice/crio-96348cfa1310aaa72d1aeab367e2bf7d8aa0dea67f1aba6376965877ad38472c WatchSource:0}: Error finding container 96348cfa1310aaa72d1aeab367e2bf7d8aa0dea67f1aba6376965877ad38472c: Status 404 returned error can't find the container with id 96348cfa1310aaa72d1aeab367e2bf7d8aa0dea67f1aba6376965877ad38472c
Sep 17 16:54:20.386235 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:20.386200    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" event={"ID":"25a12ba1-7ddb-4287-8607-e153d0e7e888","Type":"ContainerStarted","Data":"955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d"}
Sep 17 16:54:20.386636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:20.386292    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" event={"ID":"25a12ba1-7ddb-4287-8607-e153d0e7e888","Type":"ContainerStarted","Data":"96348cfa1310aaa72d1aeab367e2bf7d8aa0dea67f1aba6376965877ad38472c"}
Sep 17 16:54:20.928769 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:20.928731    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:54:21.078763 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:21.078724    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:54:23.403091 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:23.403058    2566 generic.go:356] "Generic (PLEG): container finished" podID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerID="63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3" exitCode=0
Sep 17 16:54:23.403467 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:23.403128    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" event={"ID":"f0afb92f-53d7-4c5f-bc34-d6823baf898f","Type":"ContainerDied","Data":"63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3"}
Sep 17 16:54:24.409680 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:24.409644    2566 generic.go:356] "Generic (PLEG): container finished" podID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerID="955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d" exitCode=0
Sep 17 16:54:24.410074 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:24.409717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" event={"ID":"25a12ba1-7ddb-4287-8607-e153d0e7e888","Type":"ContainerDied","Data":"955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d"}
Sep 17 16:54:25.417138 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.417104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" event={"ID":"25a12ba1-7ddb-4287-8607-e153d0e7e888","Type":"ContainerStarted","Data":"a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66"}
Sep 17 16:54:25.417567 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.417168    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" event={"ID":"25a12ba1-7ddb-4287-8607-e153d0e7e888","Type":"ContainerStarted","Data":"fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502"}
Sep 17 16:54:25.417567 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.417194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" event={"ID":"25a12ba1-7ddb-4287-8607-e153d0e7e888","Type":"ContainerStarted","Data":"15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac"}
Sep 17 16:54:25.417567 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.417458    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:25.417693 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.417632    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:25.417693 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.417680    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:25.418981 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.418948    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:54:25.419592 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.419567    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:25.439278 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.439225    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podStartSLOduration=3.449667951 podStartE2EDuration="7.439213172s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.989545221s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:54:25.436920559 +0000 UTC m=+774.148927829" watchObservedRunningTime="2026-09-17 16:54:25.439213172 +0000 UTC m=+774.151220440"
Sep 17 16:54:25.928700 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.928655    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:54:25.935079 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.935056    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:54:25.935458 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:25.935435    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:26.422243 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:26.422202    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:54:26.422746 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:26.422640    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:30.928033 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:30.927996    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:54:30.928412 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:30.928121    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:54:31.079100 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:31.079074    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:54:31.426624 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:31.426597    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:54:31.427404 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:31.427371    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:54:31.427561 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:31.427537    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:34.543586 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:34.543555    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"]
Sep 17 16:54:34.543989 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:34.543777    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container" containerID="cri-o://6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb" gracePeriod=30
Sep 17 16:54:34.543989 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:34.543806    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kube-rbac-proxy" containerID="cri-o://81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9" gracePeriod=30
Sep 17 16:54:34.564993 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:34.564969    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:54:35.469588 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:35.469552    2566 generic.go:356] "Generic (PLEG): container finished" podID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerID="81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9" exitCode=2
Sep 17 16:54:35.469742 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:35.469625    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" event={"ID":"73696000-2f51-43d4-8c56-5a4c22fa81ce","Type":"ContainerDied","Data":"81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9"}
Sep 17 16:54:35.928513 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:35.928456    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:54:35.935918 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:35.935881    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:54:35.936243 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:35.936218    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:36.073018 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:36.072978    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 17 16:54:39.087057 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.087035    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:54:39.124962 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.124933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-cabundle-cert\") pod \"73696000-2f51-43d4-8c56-5a4c22fa81ce\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") "
Sep 17 16:54:39.125105 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.125049    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-rbac-proxy-sar-config\") pod \"73696000-2f51-43d4-8c56-5a4c22fa81ce\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") "
Sep 17 16:54:39.125105 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.125078    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhzpd\" (UniqueName: \"kubernetes.io/projected/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-api-access-qhzpd\") pod \"73696000-2f51-43d4-8c56-5a4c22fa81ce\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") "
Sep 17 16:54:39.125243 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.125123    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73696000-2f51-43d4-8c56-5a4c22fa81ce-kserve-provision-location\") pod \"73696000-2f51-43d4-8c56-5a4c22fa81ce\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") "
Sep 17 16:54:39.125243 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.125172    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73696000-2f51-43d4-8c56-5a4c22fa81ce-proxy-tls\") pod \"73696000-2f51-43d4-8c56-5a4c22fa81ce\" (UID: \"73696000-2f51-43d4-8c56-5a4c22fa81ce\") "
Sep 17 16:54:39.125386 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.125339    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73696000-2f51-43d4-8c56-5a4c22fa81ce" (UID: "73696000-2f51-43d4-8c56-5a4c22fa81ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:54:39.125596 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.125478    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73696000-2f51-43d4-8c56-5a4c22fa81ce" (UID: "73696000-2f51-43d4-8c56-5a4c22fa81ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:54:39.125596 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.125523    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73696000-2f51-43d4-8c56-5a4c22fa81ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73696000-2f51-43d4-8c56-5a4c22fa81ce" (UID: "73696000-2f51-43d4-8c56-5a4c22fa81ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:54:39.125736 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.125629    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:54:39.125736 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.125648    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:54:39.125736 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.125661    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73696000-2f51-43d4-8c56-5a4c22fa81ce-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:54:39.127824 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.127798    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73696000-2f51-43d4-8c56-5a4c22fa81ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73696000-2f51-43d4-8c56-5a4c22fa81ce" (UID: "73696000-2f51-43d4-8c56-5a4c22fa81ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:54:39.127824 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.127815    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-api-access-qhzpd" (OuterVolumeSpecName: "kube-api-access-qhzpd") pod "73696000-2f51-43d4-8c56-5a4c22fa81ce" (UID: "73696000-2f51-43d4-8c56-5a4c22fa81ce"). InnerVolumeSpecName "kube-api-access-qhzpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:54:39.226911 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.226833    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhzpd\" (UniqueName: \"kubernetes.io/projected/73696000-2f51-43d4-8c56-5a4c22fa81ce-kube-api-access-qhzpd\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:54:39.226911 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.226871    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73696000-2f51-43d4-8c56-5a4c22fa81ce-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:54:39.492240 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.492204    2566 generic.go:356] "Generic (PLEG): container finished" podID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerID="6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb" exitCode=0
Sep 17 16:54:39.492418 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.492314    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"
Sep 17 16:54:39.492418 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.492320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" event={"ID":"73696000-2f51-43d4-8c56-5a4c22fa81ce","Type":"ContainerDied","Data":"6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb"}
Sep 17 16:54:39.492418 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.492392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc" event={"ID":"73696000-2f51-43d4-8c56-5a4c22fa81ce","Type":"ContainerDied","Data":"427210ff58592ab92f1507c61ba2ff8fe6e1342c2a38cf773664e50f36913ae4"}
Sep 17 16:54:39.492577 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.492433    2566 scope.go:117] "RemoveContainer" containerID="81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9"
Sep 17 16:54:39.502420 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.502399    2566 scope.go:117] "RemoveContainer" containerID="6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb"
Sep 17 16:54:39.509736 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.509717    2566 scope.go:117] "RemoveContainer" containerID="03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e"
Sep 17 16:54:39.515100 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.515081    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"]
Sep 17 16:54:39.517540 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.517524    2566 scope.go:117] "RemoveContainer" containerID="81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9"
Sep 17 16:54:39.517776 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:54:39.517757    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9\": container with ID starting with 81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9 not found: ID does not exist" containerID="81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9"
Sep 17 16:54:39.517823 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.517785    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9"} err="failed to get container status \"81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9\": rpc error: code = NotFound desc = could not find container \"81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9\": container with ID starting with 81f241e4f2348b95ed9f0652ab44d2eb353b861b42b5cc1dc5fcca122c0e0ed9 not found: ID does not exist"
Sep 17 16:54:39.517823 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.517801    2566 scope.go:117] "RemoveContainer" containerID="6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb"
Sep 17 16:54:39.518062 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:54:39.518045    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb\": container with ID starting with 6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb not found: ID does not exist" containerID="6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb"
Sep 17 16:54:39.518113 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.518066    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb"} err="failed to get container status \"6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb\": rpc error: code = NotFound desc = could not find container \"6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb\": container with ID starting with 6031d93c6ad5fb000a397cc10e0652bc88a39ff5278ad66470613b7fc5aa2fdb not found: ID does not exist"
Sep 17 16:54:39.518113 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.518081    2566 scope.go:117] "RemoveContainer" containerID="03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e"
Sep 17 16:54:39.518336 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:54:39.518318    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e\": container with ID starting with 03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e not found: ID does not exist" containerID="03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e"
Sep 17 16:54:39.518400 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.518337    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e"} err="failed to get container status \"03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e\": rpc error: code = NotFound desc = could not find container \"03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e\": container with ID starting with 03de077745ccd3cb1fcb799b515b27f318ae8c9ca2aa95b22d6e0855d1113f7e not found: ID does not exist"
Sep 17 16:54:39.521914 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.521891    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6h8gc"]
Sep 17 16:54:39.690621 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.690579    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:54:39.809884 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:39.809810    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" path="/var/lib/kubelet/pods/73696000-2f51-43d4-8c56-5a4c22fa81ce/volumes"
Sep 17 16:54:40.178415 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:40.178327    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:54:40.928783 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:40.928742    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:54:41.427582 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:41.427541    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:54:41.428047 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:41.427952    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:45.929032 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:45.928995    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:54:45.935480 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:45.935449    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:54:45.935628 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:45.935566    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:54:45.935819 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:45.935793    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:45.935908 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:45.935893    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:54:49.007171 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.007147    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:54:49.105350 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.105323    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kserve-provision-location\") pod \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") "
Sep 17 16:54:49.105485 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.105359    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0afb92f-53d7-4c5f-bc34-d6823baf898f-proxy-tls\") pod \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") "
Sep 17 16:54:49.105485 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.105377    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-cabundle-cert\") pod \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") "
Sep 17 16:54:49.105485 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.105433    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q75gl\" (UniqueName: \"kubernetes.io/projected/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-api-access-q75gl\") pod \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") "
Sep 17 16:54:49.105611 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.105497    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-rbac-proxy-sar-config\") pod \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\" (UID: \"f0afb92f-53d7-4c5f-bc34-d6823baf898f\") "
Sep 17 16:54:49.105709 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.105679    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0afb92f-53d7-4c5f-bc34-d6823baf898f" (UID: "f0afb92f-53d7-4c5f-bc34-d6823baf898f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:54:49.105880 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.105784    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f0afb92f-53d7-4c5f-bc34-d6823baf898f" (UID: "f0afb92f-53d7-4c5f-bc34-d6823baf898f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:54:49.105979 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.105908    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0afb92f-53d7-4c5f-bc34-d6823baf898f" (UID: "f0afb92f-53d7-4c5f-bc34-d6823baf898f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:54:49.107337 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.107311    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0afb92f-53d7-4c5f-bc34-d6823baf898f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0afb92f-53d7-4c5f-bc34-d6823baf898f" (UID: "f0afb92f-53d7-4c5f-bc34-d6823baf898f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:54:49.107426 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.107407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-api-access-q75gl" (OuterVolumeSpecName: "kube-api-access-q75gl") pod "f0afb92f-53d7-4c5f-bc34-d6823baf898f" (UID: "f0afb92f-53d7-4c5f-bc34-d6823baf898f"). InnerVolumeSpecName "kube-api-access-q75gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:54:49.206365 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.206297    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:54:49.206365 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.206321    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0afb92f-53d7-4c5f-bc34-d6823baf898f-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:54:49.206365 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.206331    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:54:49.206365 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.206339    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q75gl\" (UniqueName: \"kubernetes.io/projected/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-api-access-q75gl\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:54:49.206365 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.206349    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0afb92f-53d7-4c5f-bc34-d6823baf898f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:54:49.545814 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.545776    2566 generic.go:356] "Generic (PLEG): container finished" podID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerID="c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6" exitCode=0
Sep 17 16:54:49.545985 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.545840    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" event={"ID":"f0afb92f-53d7-4c5f-bc34-d6823baf898f","Type":"ContainerDied","Data":"c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6"}
Sep 17 16:54:49.545985 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.545898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt" event={"ID":"f0afb92f-53d7-4c5f-bc34-d6823baf898f","Type":"ContainerDied","Data":"dd3fad1197d4183554af1f32d39fd591a26487fcad24e6b473bf8cd76227236b"}
Sep 17 16:54:49.545985 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.545929    2566 scope.go:117] "RemoveContainer" containerID="c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6"
Sep 17 16:54:49.546115 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.545843    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"
Sep 17 16:54:49.556382 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.556357    2566 scope.go:117] "RemoveContainer" containerID="81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5"
Sep 17 16:54:49.564957 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.564938    2566 scope.go:117] "RemoveContainer" containerID="63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3"
Sep 17 16:54:49.572860 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.572843    2566 scope.go:117] "RemoveContainer" containerID="1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa"
Sep 17 16:54:49.573076 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.573056    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"]
Sep 17 16:54:49.577470 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.577445    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-755dbf648-h8qwt"]
Sep 17 16:54:49.580358 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.580345    2566 scope.go:117] "RemoveContainer" containerID="c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6"
Sep 17 16:54:49.580594 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:54:49.580574    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6\": container with ID starting with c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6 not found: ID does not exist" containerID="c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6"
Sep 17 16:54:49.580657 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.580600    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6"} err="failed to get container status \"c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6\": rpc error: code = NotFound desc = could not find container \"c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6\": container with ID starting with c86e364f354ef570b197ce72903f23b8aa88915a5af62eca1bd2ee91dd9e9cd6 not found: ID does not exist"
Sep 17 16:54:49.580657 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.580616    2566 scope.go:117] "RemoveContainer" containerID="81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5"
Sep 17 16:54:49.580881 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:54:49.580861    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5\": container with ID starting with 81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5 not found: ID does not exist" containerID="81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5"
Sep 17 16:54:49.580930 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.580890    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5"} err="failed to get container status \"81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5\": rpc error: code = NotFound desc = could not find container \"81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5\": container with ID starting with 81227075e61a46e42013a6e063b3ac835535e37ba00543f58125ebb7422835c5 not found: ID does not exist"
Sep 17 16:54:49.580930 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.580908    2566 scope.go:117] "RemoveContainer" containerID="63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3"
Sep 17 16:54:49.581139 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:54:49.581120    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3\": container with ID starting with 63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3 not found: ID does not exist" containerID="63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3"
Sep 17 16:54:49.581180 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.581144    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3"} err="failed to get container status \"63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3\": rpc error: code = NotFound desc = could not find container \"63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3\": container with ID starting with 63adc5bb87e0a5da3dd9e9c97d29123c6fe1315f08437d9d29ff40103a60fdc3 not found: ID does not exist"
Sep 17 16:54:49.581180 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.581158    2566 scope.go:117] "RemoveContainer" containerID="1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa"
Sep 17 16:54:49.581413 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:54:49.581394    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa\": container with ID starting with 1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa not found: ID does not exist" containerID="1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa"
Sep 17 16:54:49.581461 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.581420    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa"} err="failed to get container status \"1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa\": rpc error: code = NotFound desc = could not find container \"1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa\": container with ID starting with 1ef8acedeec622d3f99ea966e0e72dc2839c38170b78c9c64d8bf996914845aa not found: ID does not exist"
Sep 17 16:54:49.818807 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:49.818727    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" path="/var/lib/kubelet/pods/f0afb92f-53d7-4c5f-bc34-d6823baf898f/volumes"
Sep 17 16:54:51.427225 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:51.427177    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:54:51.427848 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:54:51.427818    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:55:01.427898 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:01.427809    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:55:01.428362 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:01.428300    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:55:11.427979 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:11.427919    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:55:11.428461 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:11.428354    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:55:21.427275 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:21.427216    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:55:21.427700 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:21.427645    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:55:31.427703 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:31.427648    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:55:31.428228 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:31.428170    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:55:41.427851 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:41.427811    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:55:41.428400 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:41.428322    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:55:54.009570 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.009537    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"]
Sep 17 16:55:54.010126 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.009796    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" containerID="cri-o://15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac" gracePeriod=30
Sep 17 16:55:54.010126 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.009853    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kube-rbac-proxy" containerID="cri-o://fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502" gracePeriod=30
Sep 17 16:55:54.010126 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.009839    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" containerID="cri-o://a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66" gracePeriod=30
Sep 17 16:55:54.033733 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.033695    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:54.107414 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.107380    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:54.112699 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.112676    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"]
Sep 17 16:55:54.113045 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113033    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="storage-initializer"
Sep 17 16:55:54.113094 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113047    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="storage-initializer"
Sep 17 16:55:54.113094 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113059    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kube-rbac-proxy"
Sep 17 16:55:54.113094 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113065    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kube-rbac-proxy"
Sep 17 16:55:54.113094 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113078    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kube-rbac-proxy"
Sep 17 16:55:54.113094 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113083    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kube-rbac-proxy"
Sep 17 16:55:54.113094 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113090    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="storage-initializer"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113096    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="storage-initializer"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113114    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113120    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113126    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113130    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113139    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113144    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113196    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kserve-container"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113204    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kserve-container"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113211    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73696000-2f51-43d4-8c56-5a4c22fa81ce" containerName="kube-rbac-proxy"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113217    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="agent"
Sep 17 16:55:54.113303 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.113223    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0afb92f-53d7-4c5f-bc34-d6823baf898f" containerName="kube-rbac-proxy"
Sep 17 16:55:54.116404 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.116383    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:54.118688 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.118663    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 17 16:55:54.118795 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.118666    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 17 16:55:54.126357 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.126334    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"]
Sep 17 16:55:54.197895 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.197868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/498a00e0-a709-4a15-a3f5-739d66788910-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-4fg2q\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:54.198008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.197917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/498a00e0-a709-4a15-a3f5-739d66788910-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-4fg2q\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:54.198008 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.197983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkhpf\" (UniqueName: \"kubernetes.io/projected/498a00e0-a709-4a15-a3f5-739d66788910-kube-api-access-rkhpf\") pod \"message-dumper-predictor-6575bcfdfd-4fg2q\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:54.298720 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.298649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/498a00e0-a709-4a15-a3f5-739d66788910-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-4fg2q\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:54.298720 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.298717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkhpf\" (UniqueName: \"kubernetes.io/projected/498a00e0-a709-4a15-a3f5-739d66788910-kube-api-access-rkhpf\") pod \"message-dumper-predictor-6575bcfdfd-4fg2q\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:54.298903 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.298786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/498a00e0-a709-4a15-a3f5-739d66788910-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-4fg2q\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:54.298903 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:55:54.298800    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 17 16:55:54.298903 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:55:54.298867    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498a00e0-a709-4a15-a3f5-739d66788910-proxy-tls podName:498a00e0-a709-4a15-a3f5-739d66788910 nodeName:}" failed. No retries permitted until 2026-09-17 16:55:54.798843068 +0000 UTC m=+863.510850326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/498a00e0-a709-4a15-a3f5-739d66788910-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-4fg2q" (UID: "498a00e0-a709-4a15-a3f5-739d66788910") : secret "message-dumper-predictor-serving-cert" not found
Sep 17 16:55:54.299393 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.299374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/498a00e0-a709-4a15-a3f5-739d66788910-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-4fg2q\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:54.307371 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.307349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkhpf\" (UniqueName: \"kubernetes.io/projected/498a00e0-a709-4a15-a3f5-739d66788910-kube-api-access-rkhpf\") pod \"message-dumper-predictor-6575bcfdfd-4fg2q\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:54.628639 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.628562    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:54.802575 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.802540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/498a00e0-a709-4a15-a3f5-739d66788910-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-4fg2q\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:54.805090 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.805057    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/498a00e0-a709-4a15-a3f5-739d66788910-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-4fg2q\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:54.856373 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.856350    2566 generic.go:356] "Generic (PLEG): container finished" podID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerID="fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502" exitCode=2
Sep 17 16:55:54.856474 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:54.856397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" event={"ID":"25a12ba1-7ddb-4287-8607-e153d0e7e888","Type":"ContainerDied","Data":"fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502"}
Sep 17 16:55:55.027591 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:55.027563    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:55.296839 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:55.296788    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"]
Sep 17 16:55:55.299016 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:55:55.298988    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498a00e0_a709_4a15_a3f5_739d66788910.slice/crio-1598ed4ef97c4dbc84386374b5e12e8f4bac7d0a565d3f9dcd49f662dde24134 WatchSource:0}: Error finding container 1598ed4ef97c4dbc84386374b5e12e8f4bac7d0a565d3f9dcd49f662dde24134: Status 404 returned error can't find the container with id 1598ed4ef97c4dbc84386374b5e12e8f4bac7d0a565d3f9dcd49f662dde24134
Sep 17 16:55:55.862115 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:55.862079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q" event={"ID":"498a00e0-a709-4a15-a3f5-739d66788910","Type":"ContainerStarted","Data":"1598ed4ef97c4dbc84386374b5e12e8f4bac7d0a565d3f9dcd49f662dde24134"}
Sep 17 16:55:56.422540 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:56.422494    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 16:55:56.869123 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:56.869088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q" event={"ID":"498a00e0-a709-4a15-a3f5-739d66788910","Type":"ContainerStarted","Data":"b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9"}
Sep 17 16:55:56.869290 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:56.869152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q" event={"ID":"498a00e0-a709-4a15-a3f5-739d66788910","Type":"ContainerStarted","Data":"91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de"}
Sep 17 16:55:56.869290 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:56.869274    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:56.869376 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:56.869325    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:56.871169 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:56.871147    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:55:56.888531 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:56.888488    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q" podStartSLOduration=1.630505871 podStartE2EDuration="2.888473598s" totalImagesPullingTime="1.257967727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:55:56.887055328 +0000 UTC m=+865.599062595" watchObservedRunningTime="2026-09-17 16:55:56.888473598 +0000 UTC m=+865.600480866"
Sep 17 16:55:58.880697 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:58.880667    2566 generic.go:356] "Generic (PLEG): container finished" podID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerID="15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac" exitCode=0
Sep 17 16:55:58.881111 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:55:58.880742    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" event={"ID":"25a12ba1-7ddb-4287-8607-e153d0e7e888","Type":"ContainerDied","Data":"15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac"}
Sep 17 16:56:01.423050 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:01.422999    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 16:56:01.427619 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:01.427586    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:56:01.428855 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:01.428826    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:56:03.885157 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:03.885128    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:56:04.116984 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.116946    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:56:04.158383 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.158308    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"]
Sep 17 16:56:04.162298 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.162277    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.164593 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.164572    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 17 16:56:04.164818 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.164803    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 17 16:56:04.173135 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.173117    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"]
Sep 17 16:56:04.278005 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.277976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-cabundle-cert\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.278113 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.278038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ecb9941-4e0b-4e71-aadf-44095637afd1-kserve-provision-location\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.278113 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.278093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ecb9941-4e0b-4e71-aadf-44095637afd1-proxy-tls\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.278203 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.278141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcp55\" (UniqueName: \"kubernetes.io/projected/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-api-access-rcp55\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.278203 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.278162    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.379090 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.379062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcp55\" (UniqueName: \"kubernetes.io/projected/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-api-access-rcp55\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.379182 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.379095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.379182 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.379130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-cabundle-cert\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.379295 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.379184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ecb9941-4e0b-4e71-aadf-44095637afd1-kserve-provision-location\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.379295 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.379219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ecb9941-4e0b-4e71-aadf-44095637afd1-proxy-tls\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.379428 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:56:04.379378    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 17 16:56:04.379494 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:56:04.379447    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ecb9941-4e0b-4e71-aadf-44095637afd1-proxy-tls podName:8ecb9941-4e0b-4e71-aadf-44095637afd1 nodeName:}" failed. No retries permitted until 2026-09-17 16:56:04.879427154 +0000 UTC m=+873.591434417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8ecb9941-4e0b-4e71-aadf-44095637afd1-proxy-tls") pod "isvc-logger-predictor-685b99d5c9-ltqlq" (UID: "8ecb9941-4e0b-4e71-aadf-44095637afd1") : secret "isvc-logger-predictor-serving-cert" not found
Sep 17 16:56:04.379657 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.379635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ecb9941-4e0b-4e71-aadf-44095637afd1-kserve-provision-location\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.379853 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.379833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-cabundle-cert\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.379906 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.379861    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.388532 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.388511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcp55\" (UniqueName: \"kubernetes.io/projected/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-api-access-rcp55\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.669351 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.669323    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:56:04.882234 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.882209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ecb9941-4e0b-4e71-aadf-44095637afd1-proxy-tls\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:04.884487 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:04.884468    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ecb9941-4e0b-4e71-aadf-44095637afd1-proxy-tls\") pod \"isvc-logger-predictor-685b99d5c9-ltqlq\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:05.072767 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:05.072735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:05.358740 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:05.352146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"]
Sep 17 16:56:05.916789 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:05.916754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" event={"ID":"8ecb9941-4e0b-4e71-aadf-44095637afd1","Type":"ContainerStarted","Data":"171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3"}
Sep 17 16:56:05.916950 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:05.916819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" event={"ID":"8ecb9941-4e0b-4e71-aadf-44095637afd1","Type":"ContainerStarted","Data":"26cfcaa042a610c3baaa846723d8f25ed5bc2f42f8f1184c9a8cbdd26e9d0cae"}
Sep 17 16:56:06.422760 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:06.422714    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 16:56:06.423194 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:06.422887    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:56:09.936749 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:09.936677    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerID="171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3" exitCode=0
Sep 17 16:56:09.937076 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:09.936748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" event={"ID":"8ecb9941-4e0b-4e71-aadf-44095637afd1","Type":"ContainerDied","Data":"171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3"}
Sep 17 16:56:10.944593 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:10.944562    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" event={"ID":"8ecb9941-4e0b-4e71-aadf-44095637afd1","Type":"ContainerStarted","Data":"5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4"}
Sep 17 16:56:10.945050 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:10.944624    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" event={"ID":"8ecb9941-4e0b-4e71-aadf-44095637afd1","Type":"ContainerStarted","Data":"0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14"}
Sep 17 16:56:10.945050 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:10.944647    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" event={"ID":"8ecb9941-4e0b-4e71-aadf-44095637afd1","Type":"ContainerStarted","Data":"9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34"}
Sep 17 16:56:10.945050 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:10.944921    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:10.945050 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:10.944976    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:10.946197 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:10.946172    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:56:10.968553 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:10.968510    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podStartSLOduration=2.8524807450000003 podStartE2EDuration="6.968495482s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.116014737s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:56:10.967525451 +0000 UTC m=+879.679532719" watchObservedRunningTime="2026-09-17 16:56:10.968495482 +0000 UTC m=+879.680502811"
Sep 17 16:56:11.423382 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:11.423346    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 16:56:11.427461 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:11.427431    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:56:11.428667 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:11.428642    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:56:11.950973 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:11.950934    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:11.950973 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:11.950949    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:56:11.952269 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:11.952216    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:56:12.955383 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:12.955347    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:56:12.955765 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:12.955585    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:56:16.422820 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:16.422785    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 16:56:17.960748 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:17.960715    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:56:17.961399 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:17.961359    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:56:17.961679 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:17.961634    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:56:21.422577 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:21.422524    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 16:56:21.428290 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:21.428226    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:56:21.428437 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:21.428362    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:5000: connect: connection refused"
Sep 17 16:56:21.428437 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:21.428384    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:56:21.428560 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:21.428463    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:56:24.161647 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.161626    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:56:24.240998 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.240939    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-cabundle-cert\") pod \"25a12ba1-7ddb-4287-8607-e153d0e7e888\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") "
Sep 17 16:56:24.240998 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.240986    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jwcn\" (UniqueName: \"kubernetes.io/projected/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-api-access-8jwcn\") pod \"25a12ba1-7ddb-4287-8607-e153d0e7e888\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") "
Sep 17 16:56:24.241172 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.241030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25a12ba1-7ddb-4287-8607-e153d0e7e888-kserve-provision-location\") pod \"25a12ba1-7ddb-4287-8607-e153d0e7e888\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") "
Sep 17 16:56:24.241172 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.241154    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-rbac-proxy-sar-config\") pod \"25a12ba1-7ddb-4287-8607-e153d0e7e888\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") "
Sep 17 16:56:24.241306 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.241231    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25a12ba1-7ddb-4287-8607-e153d0e7e888-proxy-tls\") pod \"25a12ba1-7ddb-4287-8607-e153d0e7e888\" (UID: \"25a12ba1-7ddb-4287-8607-e153d0e7e888\") "
Sep 17 16:56:24.241306 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.241246    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a12ba1-7ddb-4287-8607-e153d0e7e888-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "25a12ba1-7ddb-4287-8607-e153d0e7e888" (UID: "25a12ba1-7ddb-4287-8607-e153d0e7e888"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:56:24.241411 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.241315    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "25a12ba1-7ddb-4287-8607-e153d0e7e888" (UID: "25a12ba1-7ddb-4287-8607-e153d0e7e888"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:56:24.241523 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.241501    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "25a12ba1-7ddb-4287-8607-e153d0e7e888" (UID: "25a12ba1-7ddb-4287-8607-e153d0e7e888"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:56:24.241572 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.241509    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25a12ba1-7ddb-4287-8607-e153d0e7e888-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:56:24.241572 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.241542    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:56:24.243070 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.243051    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-api-access-8jwcn" (OuterVolumeSpecName: "kube-api-access-8jwcn") pod "25a12ba1-7ddb-4287-8607-e153d0e7e888" (UID: "25a12ba1-7ddb-4287-8607-e153d0e7e888"). InnerVolumeSpecName "kube-api-access-8jwcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:56:24.243145 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.243108    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25a12ba1-7ddb-4287-8607-e153d0e7e888-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25a12ba1-7ddb-4287-8607-e153d0e7e888" (UID: "25a12ba1-7ddb-4287-8607-e153d0e7e888"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:56:24.342723 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.342674    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25a12ba1-7ddb-4287-8607-e153d0e7e888-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:56:24.342723 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.342692    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jwcn\" (UniqueName: \"kubernetes.io/projected/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-api-access-8jwcn\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:56:24.342723 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:24.342701    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25a12ba1-7ddb-4287-8607-e153d0e7e888-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:56:25.015857 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.015826    2566 generic.go:356] "Generic (PLEG): container finished" podID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerID="a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66" exitCode=0
Sep 17 16:56:25.016012 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.015884    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"
Sep 17 16:56:25.016012 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.015888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" event={"ID":"25a12ba1-7ddb-4287-8607-e153d0e7e888","Type":"ContainerDied","Data":"a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66"}
Sep 17 16:56:25.016095 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.016012    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q" event={"ID":"25a12ba1-7ddb-4287-8607-e153d0e7e888","Type":"ContainerDied","Data":"96348cfa1310aaa72d1aeab367e2bf7d8aa0dea67f1aba6376965877ad38472c"}
Sep 17 16:56:25.016095 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.016042    2566 scope.go:117] "RemoveContainer" containerID="a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66"
Sep 17 16:56:25.028904 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.028886    2566 scope.go:117] "RemoveContainer" containerID="fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502"
Sep 17 16:56:25.036431 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.036414    2566 scope.go:117] "RemoveContainer" containerID="15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac"
Sep 17 16:56:25.040435 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.040413    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"]
Sep 17 16:56:25.044598 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.044577    2566 scope.go:117] "RemoveContainer" containerID="955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d"
Sep 17 16:56:25.044792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.044774    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-c64cf68f4-6vc6q"]
Sep 17 16:56:25.054086 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.054069    2566 scope.go:117] "RemoveContainer" containerID="a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66"
Sep 17 16:56:25.054369 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:56:25.054352    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66\": container with ID starting with a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66 not found: ID does not exist" containerID="a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66"
Sep 17 16:56:25.054416 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.054378    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66"} err="failed to get container status \"a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66\": rpc error: code = NotFound desc = could not find container \"a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66\": container with ID starting with a5685a6d388ff3151bd14748bc7e211eb23d064157093c9e544cc9f727e1ca66 not found: ID does not exist"
Sep 17 16:56:25.054416 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.054394    2566 scope.go:117] "RemoveContainer" containerID="fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502"
Sep 17 16:56:25.054657 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:56:25.054640    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502\": container with ID starting with fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502 not found: ID does not exist" containerID="fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502"
Sep 17 16:56:25.054718 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.054666    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502"} err="failed to get container status \"fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502\": rpc error: code = NotFound desc = could not find container \"fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502\": container with ID starting with fe017cfc75e69b7b1ba442f41aabe68aff9cf8974f8443b1cb8c91997a162502 not found: ID does not exist"
Sep 17 16:56:25.054718 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.054689    2566 scope.go:117] "RemoveContainer" containerID="15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac"
Sep 17 16:56:25.054964 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:56:25.054948    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac\": container with ID starting with 15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac not found: ID does not exist" containerID="15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac"
Sep 17 16:56:25.055035 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.054973    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac"} err="failed to get container status \"15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac\": rpc error: code = NotFound desc = could not find container \"15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac\": container with ID starting with 15e929fa7d7284931d7b930568ae338c7990f8bbf34501ed471d3db4279d03ac not found: ID does not exist"
Sep 17 16:56:25.055035 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.054992    2566 scope.go:117] "RemoveContainer" containerID="955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d"
Sep 17 16:56:25.055209 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:56:25.055194    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d\": container with ID starting with 955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d not found: ID does not exist" containerID="955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d"
Sep 17 16:56:25.055277 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.055214    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d"} err="failed to get container status \"955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d\": rpc error: code = NotFound desc = could not find container \"955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d\": container with ID starting with 955b2170bae637bde180d50ecc7b452079c8c9675ac1e902bf15ec0fdd06257d not found: ID does not exist"
Sep 17 16:56:25.810581 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:25.810540    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" path="/var/lib/kubelet/pods/25a12ba1-7ddb-4287-8607-e153d0e7e888/volumes"
Sep 17 16:56:27.961653 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:27.961615    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:56:27.962152 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:27.962128    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:56:30.238926 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:30.238883    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:56:35.361312 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:35.361278    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:56:35.802954 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:35.802922    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:56:37.961585 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:37.961546    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:56:37.962036 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:37.962012    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:56:47.961603 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:47.961561    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:56:47.962044 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:47.961972    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:56:57.961299 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:57.961224    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:56:57.963729 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:56:57.961672    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:57:07.961361 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:07.961320    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:57:07.961857 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:07.961826    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:57:17.961411 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:17.961365    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:57:17.961869 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:17.961831    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:57:27.962359 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:27.962326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:57:27.962732 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:27.962418    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:57:34.551288 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:34.551232    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:39.143111 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.143078    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-4fg2q_498a00e0-a709-4a15-a3f5-739d66788910/kserve-container/0.log"
Sep 17 16:57:39.304691 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.304659    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"]
Sep 17 16:57:39.305023 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.304989    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" containerID="cri-o://9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34" gracePeriod=30
Sep 17 16:57:39.305216 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.305013    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" containerID="cri-o://5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4" gracePeriod=30
Sep 17 16:57:39.305461 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.305033    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kube-rbac-proxy" containerID="cri-o://0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14" gracePeriod=30
Sep 17 16:57:39.339480 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.339454    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"]
Sep 17 16:57:39.339682 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.339652    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q" podUID="498a00e0-a709-4a15-a3f5-739d66788910" containerName="kserve-container" containerID="cri-o://91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de" gracePeriod=30
Sep 17 16:57:39.339822 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.339685    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q" podUID="498a00e0-a709-4a15-a3f5-739d66788910" containerName="kube-rbac-proxy" containerID="cri-o://b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9" gracePeriod=30
Sep 17 16:57:39.374895 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.374858    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:39.441383 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.441310    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:39.586777 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.586753    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:57:39.723415 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.723347    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkhpf\" (UniqueName: \"kubernetes.io/projected/498a00e0-a709-4a15-a3f5-739d66788910-kube-api-access-rkhpf\") pod \"498a00e0-a709-4a15-a3f5-739d66788910\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") "
Sep 17 16:57:39.723540 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.723411    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/498a00e0-a709-4a15-a3f5-739d66788910-kube-rbac-proxy-sar-config\") pod \"498a00e0-a709-4a15-a3f5-739d66788910\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") "
Sep 17 16:57:39.723540 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.723474    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/498a00e0-a709-4a15-a3f5-739d66788910-proxy-tls\") pod \"498a00e0-a709-4a15-a3f5-739d66788910\" (UID: \"498a00e0-a709-4a15-a3f5-739d66788910\") "
Sep 17 16:57:39.723809 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.723778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498a00e0-a709-4a15-a3f5-739d66788910-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "498a00e0-a709-4a15-a3f5-739d66788910" (UID: "498a00e0-a709-4a15-a3f5-739d66788910"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:57:39.725573 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.725542    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/498a00e0-a709-4a15-a3f5-739d66788910-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "498a00e0-a709-4a15-a3f5-739d66788910" (UID: "498a00e0-a709-4a15-a3f5-739d66788910"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:57:39.725668 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.725563    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498a00e0-a709-4a15-a3f5-739d66788910-kube-api-access-rkhpf" (OuterVolumeSpecName: "kube-api-access-rkhpf") pod "498a00e0-a709-4a15-a3f5-739d66788910" (UID: "498a00e0-a709-4a15-a3f5-739d66788910"). InnerVolumeSpecName "kube-api-access-rkhpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:57:39.733986 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.733963    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:39.824001 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.823983    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/498a00e0-a709-4a15-a3f5-739d66788910-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:57:39.824088 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.824002    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/498a00e0-a709-4a15-a3f5-739d66788910-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:57:39.824088 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:39.824013    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkhpf\" (UniqueName: \"kubernetes.io/projected/498a00e0-a709-4a15-a3f5-739d66788910-kube-api-access-rkhpf\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:57:40.373836 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.373806    2566 generic.go:356] "Generic (PLEG): container finished" podID="498a00e0-a709-4a15-a3f5-739d66788910" containerID="b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9" exitCode=2
Sep 17 16:57:40.373836 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.373830    2566 generic.go:356] "Generic (PLEG): container finished" podID="498a00e0-a709-4a15-a3f5-739d66788910" containerID="91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de" exitCode=2
Sep 17 16:57:40.374247 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.373854    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"
Sep 17 16:57:40.374247 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.373895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q" event={"ID":"498a00e0-a709-4a15-a3f5-739d66788910","Type":"ContainerDied","Data":"b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9"}
Sep 17 16:57:40.374247 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.373949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q" event={"ID":"498a00e0-a709-4a15-a3f5-739d66788910","Type":"ContainerDied","Data":"91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de"}
Sep 17 16:57:40.374247 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.373971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q" event={"ID":"498a00e0-a709-4a15-a3f5-739d66788910","Type":"ContainerDied","Data":"1598ed4ef97c4dbc84386374b5e12e8f4bac7d0a565d3f9dcd49f662dde24134"}
Sep 17 16:57:40.374247 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.374006    2566 scope.go:117] "RemoveContainer" containerID="b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9"
Sep 17 16:57:40.377028 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.377004    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerID="0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14" exitCode=2
Sep 17 16:57:40.377131 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.377047    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" event={"ID":"8ecb9941-4e0b-4e71-aadf-44095637afd1","Type":"ContainerDied","Data":"0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14"}
Sep 17 16:57:40.384586 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.384569    2566 scope.go:117] "RemoveContainer" containerID="91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de"
Sep 17 16:57:40.392376 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.392357    2566 scope.go:117] "RemoveContainer" containerID="b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9"
Sep 17 16:57:40.392629 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:57:40.392609    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9\": container with ID starting with b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9 not found: ID does not exist" containerID="b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9"
Sep 17 16:57:40.392709 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.392642    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"]
Sep 17 16:57:40.392709 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.392642    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9"} err="failed to get container status \"b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9\": rpc error: code = NotFound desc = could not find container \"b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9\": container with ID starting with b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9 not found: ID does not exist"
Sep 17 16:57:40.392709 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.392688    2566 scope.go:117] "RemoveContainer" containerID="91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de"
Sep 17 16:57:40.392992 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:57:40.392976    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de\": container with ID starting with 91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de not found: ID does not exist" containerID="91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de"
Sep 17 16:57:40.393050 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.392994    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de"} err="failed to get container status \"91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de\": rpc error: code = NotFound desc = could not find container \"91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de\": container with ID starting with 91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de not found: ID does not exist"
Sep 17 16:57:40.393050 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.393006    2566 scope.go:117] "RemoveContainer" containerID="b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9"
Sep 17 16:57:40.393203 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.393189    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9"} err="failed to get container status \"b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9\": rpc error: code = NotFound desc = could not find container \"b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9\": container with ID starting with b41deaf8065eed0b85fcdd9570ad8c6fdfb7ceaa2f2ae0bb2beedbc0c76417e9 not found: ID does not exist"
Sep 17 16:57:40.393268 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.393203    2566 scope.go:117] "RemoveContainer" containerID="91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de"
Sep 17 16:57:40.393419 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.393404    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de"} err="failed to get container status \"91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de\": rpc error: code = NotFound desc = could not find container \"91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de\": container with ID starting with 91c9876f318f4fe9b49746d17804abd7f3ad7a64c0818c6455f0f48a057710de not found: ID does not exist"
Sep 17 16:57:40.396592 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:40.396571    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4fg2q"]
Sep 17 16:57:41.811572 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:41.811543    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="498a00e0-a709-4a15-a3f5-739d66788910" path="/var/lib/kubelet/pods/498a00e0-a709-4a15-a3f5-739d66788910/volumes"
Sep 17 16:57:42.955836 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:42.955793    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 16:57:43.394865 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:43.394835    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerID="9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34" exitCode=0
Sep 17 16:57:43.394973 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:43.394904    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" event={"ID":"8ecb9941-4e0b-4e71-aadf-44095637afd1","Type":"ContainerDied","Data":"9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34"}
Sep 17 16:57:47.956356 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:47.956300    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 16:57:47.961629 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:47.961600    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:57:47.962079 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:47.962049    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:57:52.956422 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:52.956375    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 16:57:52.956808 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:52.956520    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:57:57.956499 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:57.956400    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 16:57:57.961794 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:57.961750    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:57:57.962211 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:57:57.962178    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:58:02.956320 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:02.956279    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 16:58:06.072792 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:06.072758    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:58:07.955616 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:07.955573    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 16:58:07.962026 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:07.961976    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:58:07.962178 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:07.962123    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:58:07.962390 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:07.962369    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:58:07.962469 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:07.962458    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:58:09.451722 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.451700    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:58:09.532032 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.532001    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerID="5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4" exitCode=0
Sep 17 16:58:09.532195 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.532054    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" event={"ID":"8ecb9941-4e0b-4e71-aadf-44095637afd1","Type":"ContainerDied","Data":"5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4"}
Sep 17 16:58:09.532195 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.532117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq" event={"ID":"8ecb9941-4e0b-4e71-aadf-44095637afd1","Type":"ContainerDied","Data":"26cfcaa042a610c3baaa846723d8f25ed5bc2f42f8f1184c9a8cbdd26e9d0cae"}
Sep 17 16:58:09.532195 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.532066    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"
Sep 17 16:58:09.532195 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.532151    2566 scope.go:117] "RemoveContainer" containerID="5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4"
Sep 17 16:58:09.540262 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.540231    2566 scope.go:117] "RemoveContainer" containerID="0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14"
Sep 17 16:58:09.548355 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.548339    2566 scope.go:117] "RemoveContainer" containerID="9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34"
Sep 17 16:58:09.552509 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.552486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ecb9941-4e0b-4e71-aadf-44095637afd1-kserve-provision-location\") pod \"8ecb9941-4e0b-4e71-aadf-44095637afd1\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") "
Sep 17 16:58:09.552607 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.552566    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-cabundle-cert\") pod \"8ecb9941-4e0b-4e71-aadf-44095637afd1\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") "
Sep 17 16:58:09.552668 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.552617    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ecb9941-4e0b-4e71-aadf-44095637afd1-proxy-tls\") pod \"8ecb9941-4e0b-4e71-aadf-44095637afd1\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") "
Sep 17 16:58:09.552724 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.552675    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcp55\" (UniqueName: \"kubernetes.io/projected/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-api-access-rcp55\") pod \"8ecb9941-4e0b-4e71-aadf-44095637afd1\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") "
Sep 17 16:58:09.552724 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.552718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-rbac-proxy-sar-config\") pod \"8ecb9941-4e0b-4e71-aadf-44095637afd1\" (UID: \"8ecb9941-4e0b-4e71-aadf-44095637afd1\") "
Sep 17 16:58:09.552826 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.552742    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb9941-4e0b-4e71-aadf-44095637afd1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ecb9941-4e0b-4e71-aadf-44095637afd1" (UID: "8ecb9941-4e0b-4e71-aadf-44095637afd1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:58:09.552954 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.552938    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ecb9941-4e0b-4e71-aadf-44095637afd1-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:58:09.553028 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.553006    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ecb9941-4e0b-4e71-aadf-44095637afd1" (UID: "8ecb9941-4e0b-4e71-aadf-44095637afd1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:58:09.553094 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.553058    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ecb9941-4e0b-4e71-aadf-44095637afd1" (UID: "8ecb9941-4e0b-4e71-aadf-44095637afd1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:58:09.554641 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.554616    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ecb9941-4e0b-4e71-aadf-44095637afd1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ecb9941-4e0b-4e71-aadf-44095637afd1" (UID: "8ecb9941-4e0b-4e71-aadf-44095637afd1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:58:09.554737 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.554719    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-api-access-rcp55" (OuterVolumeSpecName: "kube-api-access-rcp55") pod "8ecb9941-4e0b-4e71-aadf-44095637afd1" (UID: "8ecb9941-4e0b-4e71-aadf-44095637afd1"). InnerVolumeSpecName "kube-api-access-rcp55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:58:09.556198 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.556179    2566 scope.go:117] "RemoveContainer" containerID="171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3"
Sep 17 16:58:09.570515 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.570500    2566 scope.go:117] "RemoveContainer" containerID="5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4"
Sep 17 16:58:09.570778 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:58:09.570755    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4\": container with ID starting with 5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4 not found: ID does not exist" containerID="5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4"
Sep 17 16:58:09.570855 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.570785    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4"} err="failed to get container status \"5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4\": rpc error: code = NotFound desc = could not find container \"5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4\": container with ID starting with 5dd2e126e2254f4f1c586ef26410cfbe5f36ff87b84672b11cdd3653f437e7e4 not found: ID does not exist"
Sep 17 16:58:09.570855 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.570802    2566 scope.go:117] "RemoveContainer" containerID="0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14"
Sep 17 16:58:09.571059 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:58:09.571043    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14\": container with ID starting with 0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14 not found: ID does not exist" containerID="0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14"
Sep 17 16:58:09.571108 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.571065    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14"} err="failed to get container status \"0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14\": rpc error: code = NotFound desc = could not find container \"0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14\": container with ID starting with 0bee67b5f586be37112d65faf3295b98adbc7a97de0f9fb81ebb9d4e17912d14 not found: ID does not exist"
Sep 17 16:58:09.571108 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.571079    2566 scope.go:117] "RemoveContainer" containerID="9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34"
Sep 17 16:58:09.571328 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:58:09.571313    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34\": container with ID starting with 9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34 not found: ID does not exist" containerID="9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34"
Sep 17 16:58:09.571386 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.571333    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34"} err="failed to get container status \"9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34\": rpc error: code = NotFound desc = could not find container \"9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34\": container with ID starting with 9e27504855e7bd5ef62f4272464211715b176d668321ce6247300d594aecbb34 not found: ID does not exist"
Sep 17 16:58:09.571386 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.571346    2566 scope.go:117] "RemoveContainer" containerID="171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3"
Sep 17 16:58:09.571558 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:58:09.571539    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3\": container with ID starting with 171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3 not found: ID does not exist" containerID="171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3"
Sep 17 16:58:09.571610 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.571567    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3"} err="failed to get container status \"171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3\": rpc error: code = NotFound desc = could not find container \"171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3\": container with ID starting with 171ca4f55912db615055d602c7fa5380fe0e1264603dd832a1fe0d4e2d8201b3 not found: ID does not exist"
Sep 17 16:58:09.653822 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.653797    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:58:09.653822 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.653819    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ecb9941-4e0b-4e71-aadf-44095637afd1-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:58:09.653945 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.653832    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcp55\" (UniqueName: \"kubernetes.io/projected/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-api-access-rcp55\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:58:09.653945 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.653844    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ecb9941-4e0b-4e71-aadf-44095637afd1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:58:09.850220 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.850190    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"]
Sep 17 16:58:09.854152 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:09.854132    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-685b99d5c9-ltqlq"]
Sep 17 16:58:11.168265 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.168211    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:58:11.204889 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.204859    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"]
Sep 17 16:58:11.205209 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205192    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent"
Sep 17 16:58:11.205287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205212    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent"
Sep 17 16:58:11.205287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205224    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="storage-initializer"
Sep 17 16:58:11.205287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205233    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="storage-initializer"
Sep 17 16:58:11.205287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205245    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent"
Sep 17 16:58:11.205287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205264    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent"
Sep 17 16:58:11.205287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205273    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="498a00e0-a709-4a15-a3f5-739d66788910" containerName="kserve-container"
Sep 17 16:58:11.205287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205278    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="498a00e0-a709-4a15-a3f5-739d66788910" containerName="kserve-container"
Sep 17 16:58:11.205287 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205287    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="498a00e0-a709-4a15-a3f5-739d66788910" containerName="kube-rbac-proxy"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205292    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="498a00e0-a709-4a15-a3f5-739d66788910" containerName="kube-rbac-proxy"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205300    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205307    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205314    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205319    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205327    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="storage-initializer"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205332    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="storage-initializer"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205337    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kube-rbac-proxy"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205342    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kube-rbac-proxy"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205356    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kube-rbac-proxy"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205361    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kube-rbac-proxy"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205435    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kserve-container"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205447    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kserve-container"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205457    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="kube-rbac-proxy"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205467    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="498a00e0-a709-4a15-a3f5-739d66788910" containerName="kserve-container"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205475    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="kube-rbac-proxy"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205486    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" containerName="agent"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205494    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="25a12ba1-7ddb-4287-8607-e153d0e7e888" containerName="agent"
Sep 17 16:58:11.205636 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.205501    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="498a00e0-a709-4a15-a3f5-739d66788910" containerName="kube-rbac-proxy"
Sep 17 16:58:11.210504 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.210484    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.213233 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.213211    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:58:11.213357 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.213238    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 17 16:58:11.213422 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.213404    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:58:11.213482 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.213457    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 16:58:11.213549 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.213491    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 16:58:11.213607 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.213548    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:58:11.214113 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.214092    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:58:11.218965 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.218943    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"]
Sep 17 16:58:11.271707 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.266818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.271707 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.266884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7rll\" (UniqueName: \"kubernetes.io/projected/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-api-access-g7rll\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.271707 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.266922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.271707 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.266988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7be2b64-7ec1-4407-9d60-b787d343a425-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.271707 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.267071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7be2b64-7ec1-4407-9d60-b787d343a425-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.367545 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.367507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.367679 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.367553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7rll\" (UniqueName: \"kubernetes.io/projected/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-api-access-g7rll\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.367679 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.367582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.367679 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.367611    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7be2b64-7ec1-4407-9d60-b787d343a425-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.367679 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.367668    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7be2b64-7ec1-4407-9d60-b787d343a425-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.367948 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:58:11.367757    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 16:58:11.367948 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:58:11.367848    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7be2b64-7ec1-4407-9d60-b787d343a425-proxy-tls podName:a7be2b64-7ec1-4407-9d60-b787d343a425 nodeName:}" failed. No retries permitted until 2026-09-17 16:58:11.867829057 +0000 UTC m=+1000.579836321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a7be2b64-7ec1-4407-9d60-b787d343a425-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" (UID: "a7be2b64-7ec1-4407-9d60-b787d343a425") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 16:58:11.368089 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.368071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7be2b64-7ec1-4407-9d60-b787d343a425-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.368190 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.368174    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.368228 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.368204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.375446 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.375423    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7rll\" (UniqueName: \"kubernetes.io/projected/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-api-access-g7rll\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.684330 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.684301    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:58:11.809751 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.809723    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ecb9941-4e0b-4e71-aadf-44095637afd1" path="/var/lib/kubelet/pods/8ecb9941-4e0b-4e71-aadf-44095637afd1/volumes"
Sep 17 16:58:11.871739 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.871712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7be2b64-7ec1-4407-9d60-b787d343a425-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:11.873969 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:11.873939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7be2b64-7ec1-4407-9d60-b787d343a425-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:12.122903 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:12.122874    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:12.401096 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:12.401072    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"]
Sep 17 16:58:12.403023 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:58:12.402994    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7be2b64_7ec1_4407_9d60_b787d343a425.slice/crio-c6ae28feb2a9e30602439d151317f2d6f536833c39b10030db814bdd28dfdd40 WatchSource:0}: Error finding container c6ae28feb2a9e30602439d151317f2d6f536833c39b10030db814bdd28dfdd40: Status 404 returned error can't find the container with id c6ae28feb2a9e30602439d151317f2d6f536833c39b10030db814bdd28dfdd40
Sep 17 16:58:12.549906 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:12.549869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" event={"ID":"a7be2b64-7ec1-4407-9d60-b787d343a425","Type":"ContainerStarted","Data":"9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07"}
Sep 17 16:58:12.550071 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:12.549925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" event={"ID":"a7be2b64-7ec1-4407-9d60-b787d343a425","Type":"ContainerStarted","Data":"c6ae28feb2a9e30602439d151317f2d6f536833c39b10030db814bdd28dfdd40"}
Sep 17 16:58:16.570154 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:16.570120    2566 generic.go:356] "Generic (PLEG): container finished" podID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerID="9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07" exitCode=0
Sep 17 16:58:16.570518 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:16.570188    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" event={"ID":"a7be2b64-7ec1-4407-9d60-b787d343a425","Type":"ContainerDied","Data":"9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07"}
Sep 17 16:58:17.576830 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:17.576792    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" event={"ID":"a7be2b64-7ec1-4407-9d60-b787d343a425","Type":"ContainerStarted","Data":"40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079"}
Sep 17 16:58:17.577265 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:17.576856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" event={"ID":"a7be2b64-7ec1-4407-9d60-b787d343a425","Type":"ContainerStarted","Data":"9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43"}
Sep 17 16:58:17.577265 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:17.577053    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:17.577265 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:17.577114    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:17.595895 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:17.595851    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" podStartSLOduration=2.935743659 podStartE2EDuration="6.595836979s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.66009332s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:58:17.594394207 +0000 UTC m=+1006.306401476" watchObservedRunningTime="2026-09-17 16:58:17.595836979 +0000 UTC m=+1006.307844246"
Sep 17 16:58:23.585821 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:23.585791    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:58:53.586880 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:58:53.586840    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:59:03.586309 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:03.586248    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:59:13.586478 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:13.586434    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:59:23.586525 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:23.586485    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:59:30.324732 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:30.324650    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:59:31.255537 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:31.255507    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"]
Sep 17 16:59:31.255758 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:31.255730    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kserve-container" containerID="cri-o://9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43" gracePeriod=30
Sep 17 16:59:31.255856 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:31.255765    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kube-rbac-proxy" containerID="cri-o://40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079" gracePeriod=30
Sep 17 16:59:31.286034 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:31.286011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:31.924622 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:31.924583    2566 generic.go:356] "Generic (PLEG): container finished" podID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerID="40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079" exitCode=2
Sep 17 16:59:31.925004 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:31.924657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" event={"ID":"a7be2b64-7ec1-4407-9d60-b787d343a425","Type":"ContainerDied","Data":"40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079"}
Sep 17 16:59:33.581164 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:33.581125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 16:59:35.701681 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.701657    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:59:35.713892 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.713862    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7be2b64-7ec1-4407-9d60-b787d343a425-kserve-provision-location\") pod \"a7be2b64-7ec1-4407-9d60-b787d343a425\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") "
Sep 17 16:59:35.713995 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.713907    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-cabundle-cert\") pod \"a7be2b64-7ec1-4407-9d60-b787d343a425\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") "
Sep 17 16:59:35.713995 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.713951    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7be2b64-7ec1-4407-9d60-b787d343a425-proxy-tls\") pod \"a7be2b64-7ec1-4407-9d60-b787d343a425\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") "
Sep 17 16:59:35.713995 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.713990    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-rbac-proxy-sar-config\") pod \"a7be2b64-7ec1-4407-9d60-b787d343a425\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") "
Sep 17 16:59:35.714096 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.714045    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7rll\" (UniqueName: \"kubernetes.io/projected/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-api-access-g7rll\") pod \"a7be2b64-7ec1-4407-9d60-b787d343a425\" (UID: \"a7be2b64-7ec1-4407-9d60-b787d343a425\") "
Sep 17 16:59:35.714227 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.714202    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7be2b64-7ec1-4407-9d60-b787d343a425-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7be2b64-7ec1-4407-9d60-b787d343a425" (UID: "a7be2b64-7ec1-4407-9d60-b787d343a425"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:59:35.714327 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.714308    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7be2b64-7ec1-4407-9d60-b787d343a425-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:59:35.714431 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.714411    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7be2b64-7ec1-4407-9d60-b787d343a425" (UID: "a7be2b64-7ec1-4407-9d60-b787d343a425"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:59:35.714525 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.714501    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7be2b64-7ec1-4407-9d60-b787d343a425" (UID: "a7be2b64-7ec1-4407-9d60-b787d343a425"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:59:35.715891 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.715866    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7be2b64-7ec1-4407-9d60-b787d343a425-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7be2b64-7ec1-4407-9d60-b787d343a425" (UID: "a7be2b64-7ec1-4407-9d60-b787d343a425"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:59:35.716387 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.716363    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-api-access-g7rll" (OuterVolumeSpecName: "kube-api-access-g7rll") pod "a7be2b64-7ec1-4407-9d60-b787d343a425" (UID: "a7be2b64-7ec1-4407-9d60-b787d343a425"). InnerVolumeSpecName "kube-api-access-g7rll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:59:35.814883 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.814811    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:59:35.814883 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.814837    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7rll\" (UniqueName: \"kubernetes.io/projected/a7be2b64-7ec1-4407-9d60-b787d343a425-kube-api-access-g7rll\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:59:35.814883 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.814849    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7be2b64-7ec1-4407-9d60-b787d343a425-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:59:35.814883 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.814862    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7be2b64-7ec1-4407-9d60-b787d343a425-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 16:59:35.946268 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.945077    2566 generic.go:356] "Generic (PLEG): container finished" podID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerID="9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43" exitCode=0
Sep 17 16:59:35.946268 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.945183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" event={"ID":"a7be2b64-7ec1-4407-9d60-b787d343a425","Type":"ContainerDied","Data":"9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43"}
Sep 17 16:59:35.946268 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.945239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn" event={"ID":"a7be2b64-7ec1-4407-9d60-b787d343a425","Type":"ContainerDied","Data":"c6ae28feb2a9e30602439d151317f2d6f536833c39b10030db814bdd28dfdd40"}
Sep 17 16:59:35.946268 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.945296    2566 scope.go:117] "RemoveContainer" containerID="40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079"
Sep 17 16:59:35.946268 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.945457    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"
Sep 17 16:59:35.955113 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.955096    2566 scope.go:117] "RemoveContainer" containerID="9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43"
Sep 17 16:59:35.963032 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.963007    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"]
Sep 17 16:59:35.963316 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.963301    2566 scope.go:117] "RemoveContainer" containerID="9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07"
Sep 17 16:59:35.966200 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.966179    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xgzjn"]
Sep 17 16:59:35.971329 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.971310    2566 scope.go:117] "RemoveContainer" containerID="40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079"
Sep 17 16:59:35.971581 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:59:35.971562    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079\": container with ID starting with 40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079 not found: ID does not exist" containerID="40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079"
Sep 17 16:59:35.971660 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.971592    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079"} err="failed to get container status \"40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079\": rpc error: code = NotFound desc = could not find container \"40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079\": container with ID starting with 40e69de34132f19ac716244f48d104860f5d59c634664e9539d69753bad53079 not found: ID does not exist"
Sep 17 16:59:35.971660 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.971609    2566 scope.go:117] "RemoveContainer" containerID="9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43"
Sep 17 16:59:35.971836 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:59:35.971818    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43\": container with ID starting with 9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43 not found: ID does not exist" containerID="9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43"
Sep 17 16:59:35.971880 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.971844    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43"} err="failed to get container status \"9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43\": rpc error: code = NotFound desc = could not find container \"9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43\": container with ID starting with 9ee4487545a769f74fafbe67c35d1e9b47460a33389fb881cc8bd9aff23f7a43 not found: ID does not exist"
Sep 17 16:59:35.971880 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.971860    2566 scope.go:117] "RemoveContainer" containerID="9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07"
Sep 17 16:59:35.972107 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:59:35.972091    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07\": container with ID starting with 9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07 not found: ID does not exist" containerID="9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07"
Sep 17 16:59:35.972159 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:35.972109    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07"} err="failed to get container status \"9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07\": rpc error: code = NotFound desc = could not find container \"9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07\": container with ID starting with 9156e49cf2d44e402138e5b7ba78838b8d45b4a93f86e2c8b84ccb3548596a07 not found: ID does not exist"
Sep 17 16:59:36.416452 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.416417    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:36.442025 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.441995    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"]
Sep 17 16:59:36.442345 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.442333    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kserve-container"
Sep 17 16:59:36.442402 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.442347    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kserve-container"
Sep 17 16:59:36.442402 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.442360    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="storage-initializer"
Sep 17 16:59:36.442402 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.442367    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="storage-initializer"
Sep 17 16:59:36.442402 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.442381    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kube-rbac-proxy"
Sep 17 16:59:36.442402 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.442386    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kube-rbac-proxy"
Sep 17 16:59:36.442553 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.442442    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kserve-container"
Sep 17 16:59:36.442553 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.442449    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" containerName="kube-rbac-proxy"
Sep 17 16:59:36.447091 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.447072    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.449498 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.449480    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 17 16:59:36.449932 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.449907    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 16:59:36.449932 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.449909    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:59:36.450095 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.449982    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 16:59:36.450095 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.450016    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:59:36.450095 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.450021    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:59:36.450591 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.450574    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:59:36.456391 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.456370    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"]
Sep 17 16:59:36.519490 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.519465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6180006d-3ffd-4272-af33-352ccb70d270-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.519599 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.519503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6180006d-3ffd-4272-af33-352ccb70d270-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.519599 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.519524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.519682 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.519650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2nc\" (UniqueName: \"kubernetes.io/projected/6180006d-3ffd-4272-af33-352ccb70d270-kube-api-access-nv2nc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.519722 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.519709    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.620378 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.620357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.620447 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.620393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6180006d-3ffd-4272-af33-352ccb70d270-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.620447 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.620424    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6180006d-3ffd-4272-af33-352ccb70d270-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.620529 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.620453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.620529 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.620491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2nc\" (UniqueName: \"kubernetes.io/projected/6180006d-3ffd-4272-af33-352ccb70d270-kube-api-access-nv2nc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.620630 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:59:36.620494    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 17 16:59:36.620630 ip-10-0-131-202 kubenswrapper[2566]: E0917 16:59:36.620595    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6180006d-3ffd-4272-af33-352ccb70d270-proxy-tls podName:6180006d-3ffd-4272-af33-352ccb70d270 nodeName:}" failed. No retries permitted until 2026-09-17 16:59:37.120575577 +0000 UTC m=+1085.832582823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6180006d-3ffd-4272-af33-352ccb70d270-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" (UID: "6180006d-3ffd-4272-af33-352ccb70d270") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 17 16:59:36.620886 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.620867    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6180006d-3ffd-4272-af33-352ccb70d270-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.621133 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.621113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.621173 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.621121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.629006 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.628987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2nc\" (UniqueName: \"kubernetes.io/projected/6180006d-3ffd-4272-af33-352ccb70d270-kube-api-access-nv2nc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:36.690471 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:36.690407    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:37.124990 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:37.124960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6180006d-3ffd-4272-af33-352ccb70d270-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:37.127456 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:37.127436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6180006d-3ffd-4272-af33-352ccb70d270-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:37.358734 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:37.358705    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:37.630677 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:37.630655    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"]
Sep 17 16:59:37.632883 ip-10-0-131-202 kubenswrapper[2566]: W0917 16:59:37.632858    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6180006d_3ffd_4272_af33_352ccb70d270.slice/crio-96decac13d4b000421c958bff2892daff0f46d48662af9f29aceaca0576165ec WatchSource:0}: Error finding container 96decac13d4b000421c958bff2892daff0f46d48662af9f29aceaca0576165ec: Status 404 returned error can't find the container with id 96decac13d4b000421c958bff2892daff0f46d48662af9f29aceaca0576165ec
Sep 17 16:59:37.809770 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:37.809726    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7be2b64-7ec1-4407-9d60-b787d343a425" path="/var/lib/kubelet/pods/a7be2b64-7ec1-4407-9d60-b787d343a425/volumes"
Sep 17 16:59:37.958421 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:37.958334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" event={"ID":"6180006d-3ffd-4272-af33-352ccb70d270","Type":"ContainerStarted","Data":"365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a"}
Sep 17 16:59:37.958421 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:37.958388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" event={"ID":"6180006d-3ffd-4272-af33-352ccb70d270","Type":"ContainerStarted","Data":"96decac13d4b000421c958bff2892daff0f46d48662af9f29aceaca0576165ec"}
Sep 17 16:59:41.987466 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:41.987431    2566 generic.go:356] "Generic (PLEG): container finished" podID="6180006d-3ffd-4272-af33-352ccb70d270" containerID="365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a" exitCode=0
Sep 17 16:59:41.987466 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:41.987467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" event={"ID":"6180006d-3ffd-4272-af33-352ccb70d270","Type":"ContainerDied","Data":"365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a"}
Sep 17 16:59:42.994211 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:42.994175    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" event={"ID":"6180006d-3ffd-4272-af33-352ccb70d270","Type":"ContainerStarted","Data":"e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0"}
Sep 17 16:59:42.994635 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:42.994233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" event={"ID":"6180006d-3ffd-4272-af33-352ccb70d270","Type":"ContainerStarted","Data":"ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a"}
Sep 17 16:59:42.994635 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:42.994576    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:42.994635 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:42.994632    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 16:59:43.014120 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:43.014074    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" podStartSLOduration=3.475450587 podStartE2EDuration="7.014061176s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.538610589s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:59:43.011985056 +0000 UTC m=+1091.723992324" watchObservedRunningTime="2026-09-17 16:59:43.014061176 +0000 UTC m=+1091.726068444"
Sep 17 16:59:49.004569 ip-10-0-131-202 kubenswrapper[2566]: I0917 16:59:49.004530    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 17:00:19.005697 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:00:19.005654    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 17:00:25.892105 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:00:25.892070    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:00:29.005604 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:00:29.005571    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 17:00:30.984962 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:00:30.984923    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:00:31.184471 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:00:31.184434    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:00:39.005437 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:00:39.005395    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 17:00:48.569245 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:00:48.569208    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:00:48.670738 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:00:48.670700    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:00:49.005759 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:00:49.005725    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 17:00:59.431303 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:00:59.431202    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 17:01:06.511368 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:06.511335    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"]
Sep 17 17:01:06.511808 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:06.511619    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kserve-container" containerID="cri-o://ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a" gracePeriod=30
Sep 17 17:01:06.511808 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:06.511665    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kube-rbac-proxy" containerID="cri-o://e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0" gracePeriod=30
Sep 17 17:01:06.526703 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:06.526674    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:01:07.390108 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:07.390075    2566 generic.go:356] "Generic (PLEG): container finished" podID="6180006d-3ffd-4272-af33-352ccb70d270" containerID="e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0" exitCode=2
Sep 17 17:01:07.390279 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:07.390116    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" event={"ID":"6180006d-3ffd-4272-af33-352ccb70d270","Type":"ContainerDied","Data":"e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0"}
Sep 17 17:01:08.999837 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:08.999800    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 17:01:09.005172 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:09.005135    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 17:01:10.851432 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.851408    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 17:01:10.947085 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.947023    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-cabundle-cert\") pod \"6180006d-3ffd-4272-af33-352ccb70d270\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") "
Sep 17 17:01:10.947085 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.947058    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv2nc\" (UniqueName: \"kubernetes.io/projected/6180006d-3ffd-4272-af33-352ccb70d270-kube-api-access-nv2nc\") pod \"6180006d-3ffd-4272-af33-352ccb70d270\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") "
Sep 17 17:01:10.947085 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.947080    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-kube-rbac-proxy-sar-config\") pod \"6180006d-3ffd-4272-af33-352ccb70d270\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") "
Sep 17 17:01:10.947348 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.947109    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6180006d-3ffd-4272-af33-352ccb70d270-kserve-provision-location\") pod \"6180006d-3ffd-4272-af33-352ccb70d270\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") "
Sep 17 17:01:10.947348 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.947274    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6180006d-3ffd-4272-af33-352ccb70d270-proxy-tls\") pod \"6180006d-3ffd-4272-af33-352ccb70d270\" (UID: \"6180006d-3ffd-4272-af33-352ccb70d270\") "
Sep 17 17:01:10.947453 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.947426    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6180006d-3ffd-4272-af33-352ccb70d270" (UID: "6180006d-3ffd-4272-af33-352ccb70d270"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:01:10.947497 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.947445    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6180006d-3ffd-4272-af33-352ccb70d270" (UID: "6180006d-3ffd-4272-af33-352ccb70d270"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:01:10.947497 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.947476    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6180006d-3ffd-4272-af33-352ccb70d270-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6180006d-3ffd-4272-af33-352ccb70d270" (UID: "6180006d-3ffd-4272-af33-352ccb70d270"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:01:10.947571 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.947552    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:01:10.947571 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.947568    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6180006d-3ffd-4272-af33-352ccb70d270-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:01:10.947644 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.947578    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6180006d-3ffd-4272-af33-352ccb70d270-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:01:10.949102 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.949080    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6180006d-3ffd-4272-af33-352ccb70d270-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6180006d-3ffd-4272-af33-352ccb70d270" (UID: "6180006d-3ffd-4272-af33-352ccb70d270"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:01:10.949218 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:10.949150    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6180006d-3ffd-4272-af33-352ccb70d270-kube-api-access-nv2nc" (OuterVolumeSpecName: "kube-api-access-nv2nc") pod "6180006d-3ffd-4272-af33-352ccb70d270" (UID: "6180006d-3ffd-4272-af33-352ccb70d270"). InnerVolumeSpecName "kube-api-access-nv2nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:01:11.048664 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.048635    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv2nc\" (UniqueName: \"kubernetes.io/projected/6180006d-3ffd-4272-af33-352ccb70d270-kube-api-access-nv2nc\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:01:11.048664 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.048658    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6180006d-3ffd-4272-af33-352ccb70d270-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:01:11.412458 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.412426    2566 generic.go:356] "Generic (PLEG): container finished" podID="6180006d-3ffd-4272-af33-352ccb70d270" containerID="ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a" exitCode=0
Sep 17 17:01:11.412594 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.412478    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"
Sep 17 17:01:11.412594 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.412510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" event={"ID":"6180006d-3ffd-4272-af33-352ccb70d270","Type":"ContainerDied","Data":"ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a"}
Sep 17 17:01:11.412594 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.412568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc" event={"ID":"6180006d-3ffd-4272-af33-352ccb70d270","Type":"ContainerDied","Data":"96decac13d4b000421c958bff2892daff0f46d48662af9f29aceaca0576165ec"}
Sep 17 17:01:11.412716 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.412605    2566 scope.go:117] "RemoveContainer" containerID="e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0"
Sep 17 17:01:11.421400 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.421382    2566 scope.go:117] "RemoveContainer" containerID="ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a"
Sep 17 17:01:11.428821 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.428799    2566 scope.go:117] "RemoveContainer" containerID="365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a"
Sep 17 17:01:11.434411 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.434393    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"]
Sep 17 17:01:11.437149 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.437130    2566 scope.go:117] "RemoveContainer" containerID="e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0"
Sep 17 17:01:11.437437 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:01:11.437418    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0\": container with ID starting with e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0 not found: ID does not exist" containerID="e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0"
Sep 17 17:01:11.437507 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.437445    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0"} err="failed to get container status \"e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0\": rpc error: code = NotFound desc = could not find container \"e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0\": container with ID starting with e75ca9c8edf2f24de2d95125c7a6238e7d43c01fefe84f4415629843a1210fd0 not found: ID does not exist"
Sep 17 17:01:11.437507 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.437463    2566 scope.go:117] "RemoveContainer" containerID="ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a"
Sep 17 17:01:11.437709 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:01:11.437690    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a\": container with ID starting with ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a not found: ID does not exist" containerID="ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a"
Sep 17 17:01:11.437778 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.437717    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a"} err="failed to get container status \"ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a\": rpc error: code = NotFound desc = could not find container \"ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a\": container with ID starting with ac287e706732aa84c818294013a68ddb8a85bffe5b41bc472ecc86d4ffc1812a not found: ID does not exist"
Sep 17 17:01:11.437778 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.437738    2566 scope.go:117] "RemoveContainer" containerID="365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a"
Sep 17 17:01:11.437977 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:01:11.437948    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a\": container with ID starting with 365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a not found: ID does not exist" containerID="365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a"
Sep 17 17:01:11.438033 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.437972    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a"} err="failed to get container status \"365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a\": rpc error: code = NotFound desc = could not find container \"365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a\": container with ID starting with 365ba83d5a4dd441dd3d47169947d29592a74f9591d858d9844616acbf5b846a not found: ID does not exist"
Sep 17 17:01:11.438033 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.438010    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-fgvjc"]
Sep 17 17:01:11.663685 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.663595    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:01:11.696481 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.696452    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"]
Sep 17 17:01:11.696790 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.696778    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kube-rbac-proxy"
Sep 17 17:01:11.696842 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.696791    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kube-rbac-proxy"
Sep 17 17:01:11.696842 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.696810    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kserve-container"
Sep 17 17:01:11.696842 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.696815    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kserve-container"
Sep 17 17:01:11.696842 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.696824    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="storage-initializer"
Sep 17 17:01:11.696842 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.696830    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="storage-initializer"
Sep 17 17:01:11.697020 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.696880    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kserve-container"
Sep 17 17:01:11.697020 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.696892    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6180006d-3ffd-4272-af33-352ccb70d270" containerName="kube-rbac-proxy"
Sep 17 17:01:11.701652 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.701633    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.703933 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.703906    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 17 17:01:11.704064 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.703950    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:01:11.704064 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.704012    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:01:11.704312 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.704293    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:01:11.704374 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.704312    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:01:11.704374 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.704293    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:01:11.704374 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.704353    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:01:11.709959 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.709939    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"]
Sep 17 17:01:11.755431 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.755405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nngkv\" (UniqueName: \"kubernetes.io/projected/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-api-access-nngkv\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.755530 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.755438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.755530 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.755471    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.755606 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.755539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f93e7703-1771-40b2-a4dc-3e76c4e704e4-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.755643 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.755598    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.810657 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.810634    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6180006d-3ffd-4272-af33-352ccb70d270" path="/var/lib/kubelet/pods/6180006d-3ffd-4272-af33-352ccb70d270/volumes"
Sep 17 17:01:11.856494 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.856475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nngkv\" (UniqueName: \"kubernetes.io/projected/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-api-access-nngkv\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.856817 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.856508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.856817 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.856539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.856817 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.856567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f93e7703-1771-40b2-a4dc-3e76c4e704e4-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.856817 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.856594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.856817 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:01:11.856762    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 17:01:11.856817 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:01:11.856819    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f93e7703-1771-40b2-a4dc-3e76c4e704e4-proxy-tls podName:f93e7703-1771-40b2-a4dc-3e76c4e704e4 nodeName:}" failed. No retries permitted until 2026-09-17 17:01:12.356802672 +0000 UTC m=+1181.068809918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f93e7703-1771-40b2-a4dc-3e76c4e704e4-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" (UID: "f93e7703-1771-40b2-a4dc-3e76c4e704e4") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 17:01:11.857032 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.856990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.857140 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.857122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.857222 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.857207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:11.867006 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:11.866982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nngkv\" (UniqueName: \"kubernetes.io/projected/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-api-access-nngkv\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:12.047577 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:12.047547    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:01:12.360383 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:12.360308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f93e7703-1771-40b2-a4dc-3e76c4e704e4-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:12.362673 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:12.362655    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f93e7703-1771-40b2-a4dc-3e76c4e704e4-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:12.613085 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:12.613019    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:12.887368 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:12.887336    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"]
Sep 17 17:01:12.889235 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:01:12.889207    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93e7703_1771_40b2_a4dc_3e76c4e704e4.slice/crio-6f9e388de614a0f2c01ad14085348e904ebbca0a552a35dcc1520100a9034924 WatchSource:0}: Error finding container 6f9e388de614a0f2c01ad14085348e904ebbca0a552a35dcc1520100a9034924: Status 404 returned error can't find the container with id 6f9e388de614a0f2c01ad14085348e904ebbca0a552a35dcc1520100a9034924
Sep 17 17:01:13.424158 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:13.424122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" event={"ID":"f93e7703-1771-40b2-a4dc-3e76c4e704e4","Type":"ContainerStarted","Data":"b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493"}
Sep 17 17:01:13.424339 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:13.424208    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" event={"ID":"f93e7703-1771-40b2-a4dc-3e76c4e704e4","Type":"ContainerStarted","Data":"6f9e388de614a0f2c01ad14085348e904ebbca0a552a35dcc1520100a9034924"}
Sep 17 17:01:17.444654 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:17.444574    2566 generic.go:356] "Generic (PLEG): container finished" podID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerID="b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493" exitCode=0
Sep 17 17:01:17.444654 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:17.444605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" event={"ID":"f93e7703-1771-40b2-a4dc-3e76c4e704e4","Type":"ContainerDied","Data":"b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493"}
Sep 17 17:01:18.451625 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:18.451588    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" event={"ID":"f93e7703-1771-40b2-a4dc-3e76c4e704e4","Type":"ContainerStarted","Data":"4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7"}
Sep 17 17:01:18.452026 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:18.451649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" event={"ID":"f93e7703-1771-40b2-a4dc-3e76c4e704e4","Type":"ContainerStarted","Data":"1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd"}
Sep 17 17:01:18.452026 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:18.451838    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:18.470234 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:18.470186    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" podStartSLOduration=3.357973084 podStartE2EDuration="7.470171262s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.112198178s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:01:18.46854444 +0000 UTC m=+1187.180551710" watchObservedRunningTime="2026-09-17 17:01:18.470171262 +0000 UTC m=+1187.182178529"
Sep 17 17:01:19.456583 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:19.456552    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:25.467136 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:25.467106    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:01:55.468465 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:01:55.468417    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 17:02:05.468109 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:05.468067    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 17:02:15.467796 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:15.467758    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 17:02:17.060051 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:17.060017    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:22.181425 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:22.181391    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:22.550224 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:22.550186    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:25.467836 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:25.467751    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 17:02:28.222762 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:28.222730    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:35.932843 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:35.932792    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:02:41.767502 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:41.767460    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"]
Sep 17 17:02:41.768074 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:41.767704    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kserve-container" containerID="cri-o://1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd" gracePeriod=30
Sep 17 17:02:41.768074 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:41.767725    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kube-rbac-proxy" containerID="cri-o://4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7" gracePeriod=30
Sep 17 17:02:41.790692 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:41.790648    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:42.864529 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:42.864495    2566 generic.go:356] "Generic (PLEG): container finished" podID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerID="4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7" exitCode=2
Sep 17 17:02:42.864895 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:42.864566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" event={"ID":"f93e7703-1771-40b2-a4dc-3e76c4e704e4","Type":"ContainerDied","Data":"4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7"}
Sep 17 17:02:45.461852 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:45.461814    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 17:02:45.468286 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:45.468235    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 17:02:46.610798 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.610772    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:02:46.803499 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.803431    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nngkv\" (UniqueName: \"kubernetes.io/projected/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-api-access-nngkv\") pod \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") "
Sep 17 17:02:46.803499 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.803474    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-rbac-proxy-sar-config\") pod \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") "
Sep 17 17:02:46.803699 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.803505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kserve-provision-location\") pod \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") "
Sep 17 17:02:46.803699 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.803576    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-cabundle-cert\") pod \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") "
Sep 17 17:02:46.803699 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.803625    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f93e7703-1771-40b2-a4dc-3e76c4e704e4-proxy-tls\") pod \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\" (UID: \"f93e7703-1771-40b2-a4dc-3e76c4e704e4\") "
Sep 17 17:02:46.803886 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.803858    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f93e7703-1771-40b2-a4dc-3e76c4e704e4" (UID: "f93e7703-1771-40b2-a4dc-3e76c4e704e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:46.803925 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.803862    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f93e7703-1771-40b2-a4dc-3e76c4e704e4" (UID: "f93e7703-1771-40b2-a4dc-3e76c4e704e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:02:46.804018 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.803991    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f93e7703-1771-40b2-a4dc-3e76c4e704e4" (UID: "f93e7703-1771-40b2-a4dc-3e76c4e704e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:02:46.805663 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.805643    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-api-access-nngkv" (OuterVolumeSpecName: "kube-api-access-nngkv") pod "f93e7703-1771-40b2-a4dc-3e76c4e704e4" (UID: "f93e7703-1771-40b2-a4dc-3e76c4e704e4"). InnerVolumeSpecName "kube-api-access-nngkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:02:46.805779 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.805749    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93e7703-1771-40b2-a4dc-3e76c4e704e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f93e7703-1771-40b2-a4dc-3e76c4e704e4" (UID: "f93e7703-1771-40b2-a4dc-3e76c4e704e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:02:46.886610 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.886580    2566 generic.go:356] "Generic (PLEG): container finished" podID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerID="1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd" exitCode=0
Sep 17 17:02:46.886735 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.886659    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"
Sep 17 17:02:46.886820 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.886657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" event={"ID":"f93e7703-1771-40b2-a4dc-3e76c4e704e4","Type":"ContainerDied","Data":"1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd"}
Sep 17 17:02:46.886820 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.886788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9" event={"ID":"f93e7703-1771-40b2-a4dc-3e76c4e704e4","Type":"ContainerDied","Data":"6f9e388de614a0f2c01ad14085348e904ebbca0a552a35dcc1520100a9034924"}
Sep 17 17:02:46.886820 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.886815    2566 scope.go:117] "RemoveContainer" containerID="4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7"
Sep 17 17:02:46.897954 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.897938    2566 scope.go:117] "RemoveContainer" containerID="1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd"
Sep 17 17:02:46.904927 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.904879    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f93e7703-1771-40b2-a4dc-3e76c4e704e4-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:02:46.904927 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.904903    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nngkv\" (UniqueName: \"kubernetes.io/projected/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-api-access-nngkv\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:02:46.904927 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.904917    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:02:46.904927 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.904930    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f93e7703-1771-40b2-a4dc-3e76c4e704e4-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:02:46.905172 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.904944    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f93e7703-1771-40b2-a4dc-3e76c4e704e4-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:02:46.907134 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.907114    2566 scope.go:117] "RemoveContainer" containerID="b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493"
Sep 17 17:02:46.915083 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.915054    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"]
Sep 17 17:02:46.915647 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.915632    2566 scope.go:117] "RemoveContainer" containerID="4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7"
Sep 17 17:02:46.915905 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:02:46.915887    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7\": container with ID starting with 4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7 not found: ID does not exist" containerID="4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7"
Sep 17 17:02:46.915971 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.915911    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7"} err="failed to get container status \"4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7\": rpc error: code = NotFound desc = could not find container \"4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7\": container with ID starting with 4d7efbb849633e987e6c4dddba9cb62f923c94b3fa2e7f2be791b61b693365f7 not found: ID does not exist"
Sep 17 17:02:46.915971 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.915927    2566 scope.go:117] "RemoveContainer" containerID="1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd"
Sep 17 17:02:46.916166 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:02:46.916151    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd\": container with ID starting with 1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd not found: ID does not exist" containerID="1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd"
Sep 17 17:02:46.916214 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.916170    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd"} err="failed to get container status \"1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd\": rpc error: code = NotFound desc = could not find container \"1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd\": container with ID starting with 1efe69ef98d1ab51670535222a981e40d1319d8dd66d5e4b5ed6b4ff98b47efd not found: ID does not exist"
Sep 17 17:02:46.916214 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.916184    2566 scope.go:117] "RemoveContainer" containerID="b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493"
Sep 17 17:02:46.916421 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:02:46.916406    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493\": container with ID starting with b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493 not found: ID does not exist" containerID="b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493"
Sep 17 17:02:46.916464 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.916424    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493"} err="failed to get container status \"b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493\": rpc error: code = NotFound desc = could not find container \"b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493\": container with ID starting with b9a361e570e2b8e873dde2fcdc8533e8d7dcc9a52f778bd49edf904477ce4493 not found: ID does not exist"
Sep 17 17:02:46.919667 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.919646    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qkdx9"]
Sep 17 17:02:46.949870 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:46.949836    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:47.161425 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:47.161345    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:47.809602 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:47.809567    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" path="/var/lib/kubelet/pods/f93e7703-1771-40b2-a4dc-3e76c4e704e4/volumes"
Sep 17 17:02:48.971561 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:02:48.971532    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:03:04.468418 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.468377    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-a67fe7b5"]
Sep 17 17:03:04.468913 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.468751    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="storage-initializer"
Sep 17 17:03:04.468913 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.468764    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="storage-initializer"
Sep 17 17:03:04.468913 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.468776    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kserve-container"
Sep 17 17:03:04.468913 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.468782    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kserve-container"
Sep 17 17:03:04.468913 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.468798    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kube-rbac-proxy"
Sep 17 17:03:04.468913 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.468804    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kube-rbac-proxy"
Sep 17 17:03:04.468913 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.468865    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kube-rbac-proxy"
Sep 17 17:03:04.468913 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.468874    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f93e7703-1771-40b2-a4dc-3e76c4e704e4" containerName="kserve-container"
Sep 17 17:03:04.471859 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.471838    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-a67fe7b5"
Sep 17 17:03:04.475643 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.475613    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-a67fe7b5"]
Sep 17 17:03:04.541435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.541396    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflb6\" (UniqueName: \"kubernetes.io/projected/20a9850d-d91d-4a91-a394-e5310180b9d1-kube-api-access-kflb6\") pod \"curl-metrics-a67fe7b5\" (UID: \"20a9850d-d91d-4a91-a394-e5310180b9d1\") " pod="kserve/curl-metrics-a67fe7b5"
Sep 17 17:03:04.641989 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.641957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflb6\" (UniqueName: \"kubernetes.io/projected/20a9850d-d91d-4a91-a394-e5310180b9d1-kube-api-access-kflb6\") pod \"curl-metrics-a67fe7b5\" (UID: \"20a9850d-d91d-4a91-a394-e5310180b9d1\") " pod="kserve/curl-metrics-a67fe7b5"
Sep 17 17:03:04.649988 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.649956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflb6\" (UniqueName: \"kubernetes.io/projected/20a9850d-d91d-4a91-a394-e5310180b9d1-kube-api-access-kflb6\") pod \"curl-metrics-a67fe7b5\" (UID: \"20a9850d-d91d-4a91-a394-e5310180b9d1\") " pod="kserve/curl-metrics-a67fe7b5"
Sep 17 17:03:04.796569 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:04.796545    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-a67fe7b5"
Sep 17 17:03:05.063237 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:05.063164    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-a67fe7b5"]
Sep 17 17:03:05.065502 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:03:05.065476    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20a9850d_d91d_4a91_a394_e5310180b9d1.slice/crio-aef9a0989d19f7fd80064ddec649f15508a3745872ac0212a12e05c7e77f6c1d WatchSource:0}: Error finding container aef9a0989d19f7fd80064ddec649f15508a3745872ac0212a12e05c7e77f6c1d: Status 404 returned error can't find the container with id aef9a0989d19f7fd80064ddec649f15508a3745872ac0212a12e05c7e77f6c1d
Sep 17 17:03:05.067495 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:05.067477    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:03:05.975069 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:05.975037    2566 generic.go:356] "Generic (PLEG): container finished" podID="20a9850d-d91d-4a91-a394-e5310180b9d1" containerID="f3bd98e3d4805c30f92b03dfd4c90d47e26f1bc81f47e58d0d52cb4d543778dd" exitCode=0
Sep 17 17:03:05.975428 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:05.975099    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-a67fe7b5" event={"ID":"20a9850d-d91d-4a91-a394-e5310180b9d1","Type":"ContainerDied","Data":"f3bd98e3d4805c30f92b03dfd4c90d47e26f1bc81f47e58d0d52cb4d543778dd"}
Sep 17 17:03:05.975428 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:05.975126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-a67fe7b5" event={"ID":"20a9850d-d91d-4a91-a394-e5310180b9d1","Type":"ContainerStarted","Data":"aef9a0989d19f7fd80064ddec649f15508a3745872ac0212a12e05c7e77f6c1d"}
Sep 17 17:03:07.104717 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:07.104694    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-a67fe7b5"
Sep 17 17:03:07.128021 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:07.128000    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-a67fe7b5_20a9850d-d91d-4a91-a394-e5310180b9d1/curl-metrics-a67fe7b5/0.log"
Sep 17 17:03:07.152525 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:07.152496    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-a67fe7b5"]
Sep 17 17:03:07.156195 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:07.156172    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-a67fe7b5"]
Sep 17 17:03:07.164802 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:07.164785    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kflb6\" (UniqueName: \"kubernetes.io/projected/20a9850d-d91d-4a91-a394-e5310180b9d1-kube-api-access-kflb6\") pod \"20a9850d-d91d-4a91-a394-e5310180b9d1\" (UID: \"20a9850d-d91d-4a91-a394-e5310180b9d1\") "
Sep 17 17:03:07.166719 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:07.166686    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a9850d-d91d-4a91-a394-e5310180b9d1-kube-api-access-kflb6" (OuterVolumeSpecName: "kube-api-access-kflb6") pod "20a9850d-d91d-4a91-a394-e5310180b9d1" (UID: "20a9850d-d91d-4a91-a394-e5310180b9d1"). InnerVolumeSpecName "kube-api-access-kflb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:03:07.265317 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:07.265245    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kflb6\" (UniqueName: \"kubernetes.io/projected/20a9850d-d91d-4a91-a394-e5310180b9d1-kube-api-access-kflb6\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:03:07.810606 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:07.810573    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a9850d-d91d-4a91-a394-e5310180b9d1" path="/var/lib/kubelet/pods/20a9850d-d91d-4a91-a394-e5310180b9d1/volumes"
Sep 17 17:03:07.986366 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:07.986336    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-a67fe7b5"
Sep 17 17:03:07.986491 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:07.986393    2566 scope.go:117] "RemoveContainer" containerID="f3bd98e3d4805c30f92b03dfd4c90d47e26f1bc81f47e58d0d52cb4d543778dd"
Sep 17 17:03:12.515388 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:12.515357    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-e904d5ac"]
Sep 17 17:03:12.515863 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:12.515734    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a9850d-d91d-4a91-a394-e5310180b9d1" containerName="curl-metrics-a67fe7b5"
Sep 17 17:03:12.515863 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:12.515746    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a9850d-d91d-4a91-a394-e5310180b9d1" containerName="curl-metrics-a67fe7b5"
Sep 17 17:03:12.515863 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:12.515819    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a9850d-d91d-4a91-a394-e5310180b9d1" containerName="curl-metrics-a67fe7b5"
Sep 17 17:03:12.521368 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:12.521341    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e904d5ac"
Sep 17 17:03:12.524746 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:12.524710    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-e904d5ac"]
Sep 17 17:03:12.608992 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:12.608964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pld8h\" (UniqueName: \"kubernetes.io/projected/9f865177-c88c-4903-b313-4c990025a362-kube-api-access-pld8h\") pod \"curl-metrics-e904d5ac\" (UID: \"9f865177-c88c-4903-b313-4c990025a362\") " pod="kserve/curl-metrics-e904d5ac"
Sep 17 17:03:12.709648 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:12.709610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pld8h\" (UniqueName: \"kubernetes.io/projected/9f865177-c88c-4903-b313-4c990025a362-kube-api-access-pld8h\") pod \"curl-metrics-e904d5ac\" (UID: \"9f865177-c88c-4903-b313-4c990025a362\") " pod="kserve/curl-metrics-e904d5ac"
Sep 17 17:03:12.717789 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:12.717758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pld8h\" (UniqueName: \"kubernetes.io/projected/9f865177-c88c-4903-b313-4c990025a362-kube-api-access-pld8h\") pod \"curl-metrics-e904d5ac\" (UID: \"9f865177-c88c-4903-b313-4c990025a362\") " pod="kserve/curl-metrics-e904d5ac"
Sep 17 17:03:12.836556 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:12.836501    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e904d5ac"
Sep 17 17:03:13.110107 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:13.110083    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-e904d5ac"]
Sep 17 17:03:13.112620 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:03:13.112583    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f865177_c88c_4903_b313_4c990025a362.slice/crio-63258ac3bab8f85fbf5c8a1a300095707665e32e1873a91c47e0d3bc6d442302 WatchSource:0}: Error finding container 63258ac3bab8f85fbf5c8a1a300095707665e32e1873a91c47e0d3bc6d442302: Status 404 returned error can't find the container with id 63258ac3bab8f85fbf5c8a1a300095707665e32e1873a91c47e0d3bc6d442302
Sep 17 17:03:13.726679 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:13.726645    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:03:14.019035 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:14.019004    2566 generic.go:356] "Generic (PLEG): container finished" podID="9f865177-c88c-4903-b313-4c990025a362" containerID="663097a2236001c8b3a3871692c9439f14d5a94e8e04283d5ced517929c34984" exitCode=0
Sep 17 17:03:14.019197 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:14.019064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-e904d5ac" event={"ID":"9f865177-c88c-4903-b313-4c990025a362","Type":"ContainerDied","Data":"663097a2236001c8b3a3871692c9439f14d5a94e8e04283d5ced517929c34984"}
Sep 17 17:03:14.019197 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:14.019097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-e904d5ac" event={"ID":"9f865177-c88c-4903-b313-4c990025a362","Type":"ContainerStarted","Data":"63258ac3bab8f85fbf5c8a1a300095707665e32e1873a91c47e0d3bc6d442302"}
Sep 17 17:03:15.149573 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:15.149545    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e904d5ac"
Sep 17 17:03:15.167303 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:15.167279    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-e904d5ac_9f865177-c88c-4903-b313-4c990025a362/curl-metrics-e904d5ac/0.log"
Sep 17 17:03:15.189181 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:15.189152    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-e904d5ac"]
Sep 17 17:03:15.192284 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:15.192264    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-e904d5ac"]
Sep 17 17:03:15.228661 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:15.228642    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pld8h\" (UniqueName: \"kubernetes.io/projected/9f865177-c88c-4903-b313-4c990025a362-kube-api-access-pld8h\") pod \"9f865177-c88c-4903-b313-4c990025a362\" (UID: \"9f865177-c88c-4903-b313-4c990025a362\") "
Sep 17 17:03:15.230596 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:15.230575    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f865177-c88c-4903-b313-4c990025a362-kube-api-access-pld8h" (OuterVolumeSpecName: "kube-api-access-pld8h") pod "9f865177-c88c-4903-b313-4c990025a362" (UID: "9f865177-c88c-4903-b313-4c990025a362"). InnerVolumeSpecName "kube-api-access-pld8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:03:15.329623 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:15.329553    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pld8h\" (UniqueName: \"kubernetes.io/projected/9f865177-c88c-4903-b313-4c990025a362-kube-api-access-pld8h\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:03:15.809524 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:15.809496    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f865177-c88c-4903-b313-4c990025a362" path="/var/lib/kubelet/pods/9f865177-c88c-4903-b313-4c990025a362/volumes"
Sep 17 17:03:16.029210 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:16.029180    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e904d5ac"
Sep 17 17:03:16.029342 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:16.029219    2566 scope.go:117] "RemoveContainer" containerID="663097a2236001c8b3a3871692c9439f14d5a94e8e04283d5ced517929c34984"
Sep 17 17:03:20.517656 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.517620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:03:20.552161 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.552125    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"]
Sep 17 17:03:20.552545 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.552511    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f865177-c88c-4903-b313-4c990025a362" containerName="curl-metrics-e904d5ac"
Sep 17 17:03:20.552545 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.552528    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f865177-c88c-4903-b313-4c990025a362" containerName="curl-metrics-e904d5ac"
Sep 17 17:03:20.552696 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.552589    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f865177-c88c-4903-b313-4c990025a362" containerName="curl-metrics-e904d5ac"
Sep 17 17:03:20.556987 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.556964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.559729 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.559703    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:03:20.559729 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.559718    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 17:03:20.559919 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.559739    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:03:20.559919 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.559751    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 17 17:03:20.559919 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.559766    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:03:20.559919 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.559881    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:03:20.560087 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.559970    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:03:20.565430 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.565408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"]
Sep 17 17:03:20.666295 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.666269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-proxy-tls\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.666418 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.666302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.666418 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.666327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-cabundle-cert\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.666502 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.666412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kserve-provision-location\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.666502 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.666467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv2s8\" (UniqueName: \"kubernetes.io/projected/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-api-access-zv2s8\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.767636 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.767608    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-cabundle-cert\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.767773 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.767652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kserve-provision-location\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.767773 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.767706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2s8\" (UniqueName: \"kubernetes.io/projected/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-api-access-zv2s8\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.767773 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.767761    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-proxy-tls\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.767941 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.767791    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.768006 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:03:20.767942    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 17 17:03:20.768056 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:03:20.768008    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-proxy-tls podName:e08fa50e-0750-4189-8209-a5bc7cbe3b9e nodeName:}" failed. No retries permitted until 2026-09-17 17:03:21.267987929 +0000 UTC m=+1309.979995189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-proxy-tls") pod "isvc-sklearn-predictor-84dfd9559b-qz26q" (UID: "e08fa50e-0750-4189-8209-a5bc7cbe3b9e") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 17 17:03:20.768145 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.768122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kserve-provision-location\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.768505 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.768485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.768555 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.768485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-cabundle-cert\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.778395 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.778367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2s8\" (UniqueName: \"kubernetes.io/projected/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-api-access-zv2s8\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:20.875959 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:20.875937    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:03:21.272626 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:21.272601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-proxy-tls\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:21.275020 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:21.274998    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-proxy-tls\") pod \"isvc-sklearn-predictor-84dfd9559b-qz26q\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:21.468943 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:21.468907    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:21.742947 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:21.742879    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"]
Sep 17 17:03:21.745341 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:03:21.745301    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08fa50e_0750_4189_8209_a5bc7cbe3b9e.slice/crio-424e33f3a7f72b8c2cd57a19c57a49f3f5c86b8c3328a2e7cf9de2558437fd41 WatchSource:0}: Error finding container 424e33f3a7f72b8c2cd57a19c57a49f3f5c86b8c3328a2e7cf9de2558437fd41: Status 404 returned error can't find the container with id 424e33f3a7f72b8c2cd57a19c57a49f3f5c86b8c3328a2e7cf9de2558437fd41
Sep 17 17:03:22.058469 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:22.058385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" event={"ID":"e08fa50e-0750-4189-8209-a5bc7cbe3b9e","Type":"ContainerStarted","Data":"7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610"}
Sep 17 17:03:22.058469 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:22.058452    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" event={"ID":"e08fa50e-0750-4189-8209-a5bc7cbe3b9e","Type":"ContainerStarted","Data":"424e33f3a7f72b8c2cd57a19c57a49f3f5c86b8c3328a2e7cf9de2558437fd41"}
Sep 17 17:03:26.081063 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:26.080981    2566 generic.go:356] "Generic (PLEG): container finished" podID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerID="7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610" exitCode=0
Sep 17 17:03:26.081063 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:26.081052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" event={"ID":"e08fa50e-0750-4189-8209-a5bc7cbe3b9e","Type":"ContainerDied","Data":"7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610"}
Sep 17 17:03:27.088090 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:27.088051    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" event={"ID":"e08fa50e-0750-4189-8209-a5bc7cbe3b9e","Type":"ContainerStarted","Data":"8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700"}
Sep 17 17:03:27.088524 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:27.088113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" event={"ID":"e08fa50e-0750-4189-8209-a5bc7cbe3b9e","Type":"ContainerStarted","Data":"a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01"}
Sep 17 17:03:27.088524 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:27.088298    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:28.093522 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:28.093490    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:28.094780 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:28.094751    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 17:03:29.098666 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:29.098631    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 17:03:34.102787 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:34.102757    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:03:34.103234 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:34.103207    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 17:03:34.122390 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:34.122339    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podStartSLOduration=10.126990519 podStartE2EDuration="14.12232502s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.995334501s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:03:27.106578538 +0000 UTC m=+1315.818585806" watchObservedRunningTime="2026-09-17 17:03:34.12232502 +0000 UTC m=+1322.834332287"
Sep 17 17:03:44.104193 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:44.104146    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 17:03:54.103805 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:03:54.103765    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 17:04:04.103984 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:04.103941    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 17:04:14.103496 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:14.103454    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 17:04:24.103591 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:24.103556    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 17:04:28.690351 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:28.690309    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:04:33.900765 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:33.900726    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:04:34.103504 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:34.103459    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 17:04:34.126814 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:34.126776    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:04:35.804628 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:35.804584    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 17:04:45.810171 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:45.810138    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:04:50.617614 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:50.617580    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"]
Sep 17 17:04:50.618128 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:50.617814    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container" containerID="cri-o://a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01" gracePeriod=30
Sep 17 17:04:50.618128 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:50.617862    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kube-rbac-proxy" containerID="cri-o://8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700" gracePeriod=30
Sep 17 17:04:50.637408 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:50.637381    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:04:51.485160 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:51.485125    2566 generic.go:356] "Generic (PLEG): container finished" podID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerID="8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700" exitCode=2
Sep 17 17:04:51.485364 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:51.485196    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" event={"ID":"e08fa50e-0750-4189-8209-a5bc7cbe3b9e","Type":"ContainerDied","Data":"8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700"}
Sep 17 17:04:54.099082 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.099029    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 17 17:04:54.768114 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.768091    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:04:54.907900 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.907832    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-proxy-tls\") pod \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") "
Sep 17 17:04:54.907900 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.907890    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv2s8\" (UniqueName: \"kubernetes.io/projected/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-api-access-zv2s8\") pod \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") "
Sep 17 17:04:54.908102 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.907946    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-rbac-proxy-sar-config\") pod \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") "
Sep 17 17:04:54.908102 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.907980    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-cabundle-cert\") pod \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") "
Sep 17 17:04:54.908102 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.908059    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kserve-provision-location\") pod \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\" (UID: \"e08fa50e-0750-4189-8209-a5bc7cbe3b9e\") "
Sep 17 17:04:54.908382 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.908347    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e08fa50e-0750-4189-8209-a5bc7cbe3b9e" (UID: "e08fa50e-0750-4189-8209-a5bc7cbe3b9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:04:54.908499 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.908411    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e08fa50e-0750-4189-8209-a5bc7cbe3b9e" (UID: "e08fa50e-0750-4189-8209-a5bc7cbe3b9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:04:54.908499 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.908427    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e08fa50e-0750-4189-8209-a5bc7cbe3b9e" (UID: "e08fa50e-0750-4189-8209-a5bc7cbe3b9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:04:54.909959 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.909936    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e08fa50e-0750-4189-8209-a5bc7cbe3b9e" (UID: "e08fa50e-0750-4189-8209-a5bc7cbe3b9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:04:54.910033 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:54.909967    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-api-access-zv2s8" (OuterVolumeSpecName: "kube-api-access-zv2s8") pod "e08fa50e-0750-4189-8209-a5bc7cbe3b9e" (UID: "e08fa50e-0750-4189-8209-a5bc7cbe3b9e"). InnerVolumeSpecName "kube-api-access-zv2s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:04:55.009582 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.009558    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:04:55.009582 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.009579    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:04:55.009723 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.009588    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv2s8\" (UniqueName: \"kubernetes.io/projected/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-api-access-zv2s8\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:04:55.009723 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.009597    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:04:55.009723 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.009606    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e08fa50e-0750-4189-8209-a5bc7cbe3b9e-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:04:55.505072 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.505036    2566 generic.go:356] "Generic (PLEG): container finished" podID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerID="a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01" exitCode=0
Sep 17 17:04:55.505483 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.505089    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"
Sep 17 17:04:55.505483 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.505102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" event={"ID":"e08fa50e-0750-4189-8209-a5bc7cbe3b9e","Type":"ContainerDied","Data":"a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01"}
Sep 17 17:04:55.505483 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.505157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q" event={"ID":"e08fa50e-0750-4189-8209-a5bc7cbe3b9e","Type":"ContainerDied","Data":"424e33f3a7f72b8c2cd57a19c57a49f3f5c86b8c3328a2e7cf9de2558437fd41"}
Sep 17 17:04:55.505483 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.505185    2566 scope.go:117] "RemoveContainer" containerID="8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700"
Sep 17 17:04:55.514102 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.514086    2566 scope.go:117] "RemoveContainer" containerID="a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01"
Sep 17 17:04:55.521164 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.521146    2566 scope.go:117] "RemoveContainer" containerID="7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610"
Sep 17 17:04:55.526622 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.526597    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"]
Sep 17 17:04:55.529291 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.529273    2566 scope.go:117] "RemoveContainer" containerID="8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700"
Sep 17 17:04:55.529537 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:04:55.529518    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700\": container with ID starting with 8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700 not found: ID does not exist" containerID="8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700"
Sep 17 17:04:55.529612 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.529549    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700"} err="failed to get container status \"8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700\": rpc error: code = NotFound desc = could not find container \"8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700\": container with ID starting with 8b02611f9c67c3628a907a72a46878557857332b2ba8bc564dec2eaf0092d700 not found: ID does not exist"
Sep 17 17:04:55.529612 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.529572    2566 scope.go:117] "RemoveContainer" containerID="a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01"
Sep 17 17:04:55.529823 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:04:55.529808    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01\": container with ID starting with a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01 not found: ID does not exist" containerID="a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01"
Sep 17 17:04:55.529884 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.529831    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01"} err="failed to get container status \"a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01\": rpc error: code = NotFound desc = could not find container \"a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01\": container with ID starting with a0e7af7f4ec1e24b4ca2ac8459a1537cddff44ef198225b5eb98c43fa3156d01 not found: ID does not exist"
Sep 17 17:04:55.529884 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.529851    2566 scope.go:117] "RemoveContainer" containerID="7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610"
Sep 17 17:04:55.530039 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.530022    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84dfd9559b-qz26q"]
Sep 17 17:04:55.530089 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:04:55.530073    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610\": container with ID starting with 7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610 not found: ID does not exist" containerID="7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610"
Sep 17 17:04:55.530124 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.530095    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610"} err="failed to get container status \"7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610\": rpc error: code = NotFound desc = could not find container \"7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610\": container with ID starting with 7d54e46104f9d30bf7d42a9562cba54363c1f5072fdfafae40a9745256e01610 not found: ID does not exist"
Sep 17 17:04:55.810284 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.810184    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" path="/var/lib/kubelet/pods/e08fa50e-0750-4189-8209-a5bc7cbe3b9e/volumes"
Sep 17 17:04:55.814452 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.814424    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:04:55.851010 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.850987    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"]
Sep 17 17:04:55.851332 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.851319    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="storage-initializer"
Sep 17 17:04:55.851391 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.851334    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="storage-initializer"
Sep 17 17:04:55.851391 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.851358    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kube-rbac-proxy"
Sep 17 17:04:55.851391 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.851364    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kube-rbac-proxy"
Sep 17 17:04:55.851391 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.851378    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container"
Sep 17 17:04:55.851391 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.851383    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container"
Sep 17 17:04:55.851566 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.851429    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kserve-container"
Sep 17 17:04:55.851566 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.851441    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e08fa50e-0750-4189-8209-a5bc7cbe3b9e" containerName="kube-rbac-proxy"
Sep 17 17:04:55.855933 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.855910    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:55.858682 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.858661    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:04:55.858855 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.858786    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:04:55.858964 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.858948    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 17 17:04:55.859031 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.858953    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 17:04:55.859031 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.859006    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:04:55.859031 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.859016    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:04:55.859031 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.859028    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:04:55.866125 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:55.866103    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"]
Sep 17 17:04:56.017709 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.017676    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd78v\" (UniqueName: \"kubernetes.io/projected/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-api-access-gd78v\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.017709 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.017711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.017958 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.017783    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.017958 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.017852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.017958 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.017953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.119128 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.119057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.119128 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.119109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd78v\" (UniqueName: \"kubernetes.io/projected/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-api-access-gd78v\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.119337 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.119223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.119337 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.119285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.119337 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.119330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.119478 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:04:56.119445    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 17 17:04:56.119535 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:04:56.119520    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-proxy-tls podName:c6d64c0c-90ea-4566-a081-a51bee1c8d3d nodeName:}" failed. No retries permitted until 2026-09-17 17:04:56.619500617 +0000 UTC m=+1405.331507887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" (UID: "c6d64c0c-90ea-4566-a081-a51bee1c8d3d") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 17 17:04:56.119681 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.119661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.119765 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.119750    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.119881 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.119863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.128281 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.128245    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd78v\" (UniqueName: \"kubernetes.io/projected/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-api-access-gd78v\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.286580 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.286551    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:04:56.623449 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.623421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.625911 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.625886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:56.769355 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:56.769332    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:04:57.045776 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:57.045754    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"]
Sep 17 17:04:57.047401 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:04:57.047373    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d64c0c_90ea_4566_a081_a51bee1c8d3d.slice/crio-cf619ddc8b4c27cf73ae1e02b464f7ac0ff68db52a52ce65a2db31004eec3d4e WatchSource:0}: Error finding container cf619ddc8b4c27cf73ae1e02b464f7ac0ff68db52a52ce65a2db31004eec3d4e: Status 404 returned error can't find the container with id cf619ddc8b4c27cf73ae1e02b464f7ac0ff68db52a52ce65a2db31004eec3d4e
Sep 17 17:04:57.517570 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:57.517530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" event={"ID":"c6d64c0c-90ea-4566-a081-a51bee1c8d3d","Type":"ContainerStarted","Data":"d95c72f848b80424e17edd9948dbb43e49356e7acb20aecf824aeee1075cfc6a"}
Sep 17 17:04:57.517752 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:04:57.517593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" event={"ID":"c6d64c0c-90ea-4566-a081-a51bee1c8d3d","Type":"ContainerStarted","Data":"cf619ddc8b4c27cf73ae1e02b464f7ac0ff68db52a52ce65a2db31004eec3d4e"}
Sep 17 17:05:01.536816 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:05:01.536783    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerID="d95c72f848b80424e17edd9948dbb43e49356e7acb20aecf824aeee1075cfc6a" exitCode=0
Sep 17 17:05:01.537187 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:05:01.536855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" event={"ID":"c6d64c0c-90ea-4566-a081-a51bee1c8d3d","Type":"ContainerDied","Data":"d95c72f848b80424e17edd9948dbb43e49356e7acb20aecf824aeee1075cfc6a"}
Sep 17 17:05:08.609381 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:05:08.609344    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:06:19.858656 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:06:19.858610    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:06:24.984218 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:06:24.984180    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:06:25.163288 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:06:25.163230    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:05.218078 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:05.218041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" event={"ID":"c6d64c0c-90ea-4566-a081-a51bee1c8d3d","Type":"ContainerStarted","Data":"1ec8f31568bd8f4d3b95ef7b2827553e723fffbd8151bbdbb300be0c750783a8"}
Sep 17 17:07:05.218578 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:05.218100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" event={"ID":"c6d64c0c-90ea-4566-a081-a51bee1c8d3d","Type":"ContainerStarted","Data":"aee8419dd299841d9b6caf43e768d0c4a1a3ec9188bcb57050614fc016815d47"}
Sep 17 17:07:05.218578 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:05.218192    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:07:05.238163 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:05.238118    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" podStartSLOduration=3.41049261 podStartE2EDuration="2m10.238105151s" totalImagesPullingTime="2m3.164334276s" totalInitContainerRuntime="3.663278265s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:07:05.23683095 +0000 UTC m=+1533.948838221" watchObservedRunningTime="2026-09-17 17:07:05.238105151 +0000 UTC m=+1533.950112419"
Sep 17 17:07:06.222808 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:06.222773    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:07:08.888644 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:08.888607    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:12.232119 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:12.232090    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:07:12.472622 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:12.472587    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:12.726584 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:12.726478    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:42.323452 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:42.323406    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 17:07:47.812377 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:47.812341    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:52.234644 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.234616    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:07:52.909356 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.909319    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:52.953812 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.953777    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"]
Sep 17 17:07:52.956722 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.956700    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:52.959077 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.959052    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 17:07:52.959077 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.959069    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 17:07:52.965437 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.965032    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"]
Sep 17 17:07:52.968176 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.968156    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"]
Sep 17 17:07:52.968322 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.968304    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:52.970417 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.970400    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 17:07:52.982873 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.982839    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"]
Sep 17 17:07:52.984662 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:52.984641    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:53.050729 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.050702    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:53.054900 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.054868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.055025 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.054924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.055025 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.054982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.055123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.055030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dda84025-b5a8-4825-bf31-2ec23ccd695b-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.055123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.055082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67gfx\" (UniqueName: \"kubernetes.io/projected/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-api-access-67gfx\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.055123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.055119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda84025-b5a8-4825-bf31-2ec23ccd695b-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.055281 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.055142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.055281 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.055172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldk29\" (UniqueName: \"kubernetes.io/projected/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-api-access-ldk29\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.055281 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.055213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.055281 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.055240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.156415 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.156385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dda84025-b5a8-4825-bf31-2ec23ccd695b-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.156529 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.156423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67gfx\" (UniqueName: \"kubernetes.io/projected/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-api-access-67gfx\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.156529 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.156442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda84025-b5a8-4825-bf31-2ec23ccd695b-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.156606 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:07:53.156537    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 17:07:53.156606 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.156575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.156703 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:07:53.156607    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda84025-b5a8-4825-bf31-2ec23ccd695b-proxy-tls podName:dda84025-b5a8-4825-bf31-2ec23ccd695b nodeName:}" failed. No retries permitted until 2026-09-17 17:07:53.656586814 +0000 UTC m=+1582.368594082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dda84025-b5a8-4825-bf31-2ec23ccd695b-proxy-tls") pod "isvc-canary-create-predictor-8cb968449-r462g" (UID: "dda84025-b5a8-4825-bf31-2ec23ccd695b") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 17:07:53.156703 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.156636    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldk29\" (UniqueName: \"kubernetes.io/projected/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-api-access-ldk29\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.156703 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.156687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.156867 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.156714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.156867 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.156780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.157291 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.156840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dda84025-b5a8-4825-bf31-2ec23ccd695b-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.157291 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.157277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.157491 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.157308    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.157491 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.157326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.157491 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.157043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.157491 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.157422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.157767 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.157748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.157857 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.157837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.159125 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.159107    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.164221 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.164166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67gfx\" (UniqueName: \"kubernetes.io/projected/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-api-access-67gfx\") pod \"isvc-canary-create-v2-predictor-7768896644-b7988\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.164487 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.164467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldk29\" (UniqueName: \"kubernetes.io/projected/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-api-access-ldk29\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.268109 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.268083    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:53.466493 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.466416    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:53.540082 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.540058    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"]
Sep 17 17:07:53.542374 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:07:53.542344    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6e48ad_d5e5_44ba_bbe2_2b07f04aabba.slice/crio-31c1cadd4d79ec97c597a0cad2ab0759f2e88c753c4165f2d59f91e0640bcca8 WatchSource:0}: Error finding container 31c1cadd4d79ec97c597a0cad2ab0759f2e88c753c4165f2d59f91e0640bcca8: Status 404 returned error can't find the container with id 31c1cadd4d79ec97c597a0cad2ab0759f2e88c753c4165f2d59f91e0640bcca8
Sep 17 17:07:53.661850 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.661818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda84025-b5a8-4825-bf31-2ec23ccd695b-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.663969 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.663946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda84025-b5a8-4825-bf31-2ec23ccd695b-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-r462g\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:53.878677 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:53.878648    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:54.151781 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:54.151756    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"]
Sep 17 17:07:54.155029 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:07:54.155002    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda84025_b5a8_4825_bf31_2ec23ccd695b.slice/crio-20bb0cbaadac2431755a620cf1400927378a0567726e4624dce6d633ff619362 WatchSource:0}: Error finding container 20bb0cbaadac2431755a620cf1400927378a0567726e4624dce6d633ff619362: Status 404 returned error can't find the container with id 20bb0cbaadac2431755a620cf1400927378a0567726e4624dce6d633ff619362
Sep 17 17:07:54.447779 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:54.447682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" event={"ID":"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba","Type":"ContainerStarted","Data":"6764dac8686974e27cc87d7096d2f0e92de06e62b5075db371b4a0e5cb88e805"}
Sep 17 17:07:54.448190 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:54.447776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" event={"ID":"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba","Type":"ContainerStarted","Data":"31c1cadd4d79ec97c597a0cad2ab0759f2e88c753c4165f2d59f91e0640bcca8"}
Sep 17 17:07:54.449469 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:54.449433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" event={"ID":"dda84025-b5a8-4825-bf31-2ec23ccd695b","Type":"ContainerStarted","Data":"bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b"}
Sep 17 17:07:54.449592 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:54.449493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" event={"ID":"dda84025-b5a8-4825-bf31-2ec23ccd695b","Type":"ContainerStarted","Data":"20bb0cbaadac2431755a620cf1400927378a0567726e4624dce6d633ff619362"}
Sep 17 17:07:55.909532 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:55.909501    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"]
Sep 17 17:07:55.909911 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:55.909750    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kserve-container" containerID="cri-o://aee8419dd299841d9b6caf43e768d0c4a1a3ec9188bcb57050614fc016815d47" gracePeriod=30
Sep 17 17:07:55.909911 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:55.909780    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kube-rbac-proxy" containerID="cri-o://1ec8f31568bd8f4d3b95ef7b2827553e723fffbd8151bbdbb300be0c750783a8" gracePeriod=30
Sep 17 17:07:55.931114 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:55.931083    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:56.461294 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:56.461245    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerID="1ec8f31568bd8f4d3b95ef7b2827553e723fffbd8151bbdbb300be0c750783a8" exitCode=2
Sep 17 17:07:56.461294 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:56.461298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" event={"ID":"c6d64c0c-90ea-4566-a081-a51bee1c8d3d","Type":"ContainerDied","Data":"1ec8f31568bd8f4d3b95ef7b2827553e723fffbd8151bbdbb300be0c750783a8"}
Sep 17 17:07:57.227770 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:57.227724    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 17:07:57.474413 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:57.474379    2566 generic.go:356] "Generic (PLEG): container finished" podID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerID="6764dac8686974e27cc87d7096d2f0e92de06e62b5075db371b4a0e5cb88e805" exitCode=0
Sep 17 17:07:57.474553 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:57.474436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" event={"ID":"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba","Type":"ContainerDied","Data":"6764dac8686974e27cc87d7096d2f0e92de06e62b5075db371b4a0e5cb88e805"}
Sep 17 17:07:58.480682 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:58.480648    2566 generic.go:356] "Generic (PLEG): container finished" podID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerID="bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b" exitCode=0
Sep 17 17:07:58.481059 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:58.480719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" event={"ID":"dda84025-b5a8-4825-bf31-2ec23ccd695b","Type":"ContainerDied","Data":"bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b"}
Sep 17 17:07:58.483103 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:58.483062    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" event={"ID":"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba","Type":"ContainerStarted","Data":"be4b8fca850b127eedc12854a0889827c89181416c47772f15e3682f2e559197"}
Sep 17 17:07:58.483169 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:58.483136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" event={"ID":"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba","Type":"ContainerStarted","Data":"5e9bf5d269b7399953a286ec904f2f4be13e8a444851a66891cff1524df940df"}
Sep 17 17:07:58.483374 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:58.483355    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:58.483469 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:58.483395    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:07:58.484784 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:58.484759    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 17:07:58.516404 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:58.516360    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podStartSLOduration=2.744738536 podStartE2EDuration="6.516346745s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.771608209s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:07:58.515347243 +0000 UTC m=+1587.227354511" watchObservedRunningTime="2026-09-17 17:07:58.516346745 +0000 UTC m=+1587.228354012"
Sep 17 17:07:59.489763 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:59.489728    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" event={"ID":"dda84025-b5a8-4825-bf31-2ec23ccd695b","Type":"ContainerStarted","Data":"75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5"}
Sep 17 17:07:59.490135 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:59.489785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" event={"ID":"dda84025-b5a8-4825-bf31-2ec23ccd695b","Type":"ContainerStarted","Data":"89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8"}
Sep 17 17:07:59.490135 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:59.489980    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 17:07:59.490240 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:59.490215    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:59.490313 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:59.490285    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:07:59.491410 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:59.491387    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 17:07:59.508837 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:07:59.508793    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podStartSLOduration=3.482891016 podStartE2EDuration="7.508781346s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.02589033s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:07:59.506846608 +0000 UTC m=+1588.218853865" watchObservedRunningTime="2026-09-17 17:07:59.508781346 +0000 UTC m=+1588.220788614"
Sep 17 17:08:00.494658 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:00.494620    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 17:08:01.056550 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.056514    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:08:01.089957 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.089920    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"]
Sep 17 17:08:01.092937 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.092913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.095278 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.095241    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 17 17:08:01.095375 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.095279    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 17 17:08:01.103820 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.103797    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"]
Sep 17 17:08:01.229073 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.229032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.229227 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.229111    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f27715f8-628e-434c-8ef8-40f2a61c5ad4-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.229340 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.229232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.229409 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.229343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqwk\" (UniqueName: \"kubernetes.io/projected/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-api-access-jkqwk\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.229409 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.229375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.330427 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.330348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.330427 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.330413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqwk\" (UniqueName: \"kubernetes.io/projected/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-api-access-jkqwk\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.330621 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.330440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.330621 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.330477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.330733 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.330627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f27715f8-628e-434c-8ef8-40f2a61c5ad4-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.330928 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.330907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.331039 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:08:01.331021    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 17 17:08:01.331106 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:08:01.331095    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f27715f8-628e-434c-8ef8-40f2a61c5ad4-proxy-tls podName:f27715f8-628e-434c-8ef8-40f2a61c5ad4 nodeName:}" failed. No retries permitted until 2026-09-17 17:08:01.831074787 +0000 UTC m=+1590.543082061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f27715f8-628e-434c-8ef8-40f2a61c5ad4-proxy-tls") pod "isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" (UID: "f27715f8-628e-434c-8ef8-40f2a61c5ad4") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 17 17:08:01.331226 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.331194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.331589 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.331568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.340052 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.340034    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqwk\" (UniqueName: \"kubernetes.io/projected/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-api-access-jkqwk\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.778487 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.778454    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:08:01.835285 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.835230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f27715f8-628e-434c-8ef8-40f2a61c5ad4-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:01.837591 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:01.837564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f27715f8-628e-434c-8ef8-40f2a61c5ad4-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7fbff8696d-wttbk\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:02.004174 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.004143    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:02.227811 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.227769    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 17:08:02.233296 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.233246    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.47:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 17:08:02.282574 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.282553    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"]
Sep 17 17:08:02.284247 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:08:02.284223    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27715f8_628e_434c_8ef8_40f2a61c5ad4.slice/crio-50e0f8f1b162f6fdfcf7f31bcbc506b742962659c63f868499fe9d4adc7c2304 WatchSource:0}: Error finding container 50e0f8f1b162f6fdfcf7f31bcbc506b742962659c63f868499fe9d4adc7c2304: Status 404 returned error can't find the container with id 50e0f8f1b162f6fdfcf7f31bcbc506b742962659c63f868499fe9d4adc7c2304
Sep 17 17:08:02.506835 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.506799    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerID="aee8419dd299841d9b6caf43e768d0c4a1a3ec9188bcb57050614fc016815d47" exitCode=0
Sep 17 17:08:02.506991 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.506883    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" event={"ID":"c6d64c0c-90ea-4566-a081-a51bee1c8d3d","Type":"ContainerDied","Data":"aee8419dd299841d9b6caf43e768d0c4a1a3ec9188bcb57050614fc016815d47"}
Sep 17 17:08:02.508945 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.508916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" event={"ID":"f27715f8-628e-434c-8ef8-40f2a61c5ad4","Type":"ContainerStarted","Data":"6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f"}
Sep 17 17:08:02.509082 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.508974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" event={"ID":"f27715f8-628e-434c-8ef8-40f2a61c5ad4","Type":"ContainerStarted","Data":"50e0f8f1b162f6fdfcf7f31bcbc506b742962659c63f868499fe9d4adc7c2304"}
Sep 17 17:08:02.563066 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.563041    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:08:02.642034 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.641966    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd78v\" (UniqueName: \"kubernetes.io/projected/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-api-access-gd78v\") pod \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") "
Sep 17 17:08:02.642034 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.642011    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-cabundle-cert\") pod \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") "
Sep 17 17:08:02.642245 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.642070    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kserve-provision-location\") pod \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") "
Sep 17 17:08:02.642245 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.642100    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-rbac-proxy-sar-config\") pod \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") "
Sep 17 17:08:02.642245 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.642131    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-proxy-tls\") pod \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\" (UID: \"c6d64c0c-90ea-4566-a081-a51bee1c8d3d\") "
Sep 17 17:08:02.642443 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.642375    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6d64c0c-90ea-4566-a081-a51bee1c8d3d" (UID: "c6d64c0c-90ea-4566-a081-a51bee1c8d3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:08:02.642443 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.642415    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6d64c0c-90ea-4566-a081-a51bee1c8d3d" (UID: "c6d64c0c-90ea-4566-a081-a51bee1c8d3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:08:02.642548 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.642463    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6d64c0c-90ea-4566-a081-a51bee1c8d3d" (UID: "c6d64c0c-90ea-4566-a081-a51bee1c8d3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:08:02.642599 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.642541    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:08:02.647357 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.642563    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:08:02.651107 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.651065    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-api-access-gd78v" (OuterVolumeSpecName: "kube-api-access-gd78v") pod "c6d64c0c-90ea-4566-a081-a51bee1c8d3d" (UID: "c6d64c0c-90ea-4566-a081-a51bee1c8d3d"). InnerVolumeSpecName "kube-api-access-gd78v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:08:02.651596 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.651574    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6d64c0c-90ea-4566-a081-a51bee1c8d3d" (UID: "c6d64c0c-90ea-4566-a081-a51bee1c8d3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:08:02.744320 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.744278    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:08:02.744320 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.744320    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:08:02.744496 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:02.744337    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd78v\" (UniqueName: \"kubernetes.io/projected/c6d64c0c-90ea-4566-a081-a51bee1c8d3d-kube-api-access-gd78v\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:08:03.515674 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:03.515641    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"
Sep 17 17:08:03.515674 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:03.515651    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8" event={"ID":"c6d64c0c-90ea-4566-a081-a51bee1c8d3d","Type":"ContainerDied","Data":"cf619ddc8b4c27cf73ae1e02b464f7ac0ff68db52a52ce65a2db31004eec3d4e"}
Sep 17 17:08:03.516187 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:03.515721    2566 scope.go:117] "RemoveContainer" containerID="1ec8f31568bd8f4d3b95ef7b2827553e723fffbd8151bbdbb300be0c750783a8"
Sep 17 17:08:03.525896 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:03.525873    2566 scope.go:117] "RemoveContainer" containerID="aee8419dd299841d9b6caf43e768d0c4a1a3ec9188bcb57050614fc016815d47"
Sep 17 17:08:03.533731 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:03.533713    2566 scope.go:117] "RemoveContainer" containerID="d95c72f848b80424e17edd9948dbb43e49356e7acb20aecf824aeee1075cfc6a"
Sep 17 17:08:03.540733 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:03.540708    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"]
Sep 17 17:08:03.543965 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:03.543948    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-8x8r8"]
Sep 17 17:08:03.810188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:03.810119    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" path="/var/lib/kubelet/pods/c6d64c0c-90ea-4566-a081-a51bee1c8d3d/volumes"
Sep 17 17:08:04.499977 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:04.499947    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:08:04.500460 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:04.500437    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 17:08:05.499351 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:05.499323    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:08:05.499823 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:05.499795    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 17:08:08.543050 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:08.543016    2566 generic.go:356] "Generic (PLEG): container finished" podID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerID="6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f" exitCode=0
Sep 17 17:08:08.543436 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:08.543086    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" event={"ID":"f27715f8-628e-434c-8ef8-40f2a61c5ad4","Type":"ContainerDied","Data":"6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f"}
Sep 17 17:08:09.550028 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:09.549998    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" event={"ID":"f27715f8-628e-434c-8ef8-40f2a61c5ad4","Type":"ContainerStarted","Data":"a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04"}
Sep 17 17:08:09.550460 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:09.550057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" event={"ID":"f27715f8-628e-434c-8ef8-40f2a61c5ad4","Type":"ContainerStarted","Data":"d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81"}
Sep 17 17:08:09.550460 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:09.550241    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:09.571318 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:09.571267    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" podStartSLOduration=2.961393568 podStartE2EDuration="8.571236841s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.609843273s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:08:09.569400379 +0000 UTC m=+1598.281407648" watchObservedRunningTime="2026-09-17 17:08:09.571236841 +0000 UTC m=+1598.283244108"
Sep 17 17:08:10.555551 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:10.555519    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:10.557176 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:10.557149    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 17:08:11.560557 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:11.560509    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 17:08:14.501214 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:14.501164    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 17:08:15.500308 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:15.500267    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 17:08:16.565636 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:16.565605    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:16.566187 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:16.566157    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 17:08:24.500894 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:24.500813    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 17:08:25.499811 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:25.499775    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 17:08:26.567268 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:26.567221    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:32.996083 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:32.996049    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"]
Sep 17 17:08:32.996576 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:32.996296    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="kserve-container" containerID="cri-o://d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81" gracePeriod=30
Sep 17 17:08:32.996576 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:32.996328    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="kube-rbac-proxy" containerID="cri-o://a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04" gracePeriod=30
Sep 17 17:08:33.020396 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:33.020365    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:08:33.678796 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:33.678768    2566 generic.go:356] "Generic (PLEG): container finished" podID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerID="a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04" exitCode=2
Sep 17 17:08:33.678935 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:33.678838    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" event={"ID":"f27715f8-628e-434c-8ef8-40f2a61c5ad4","Type":"ContainerDied","Data":"a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04"}
Sep 17 17:08:33.942573 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:33.942552    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:34.026602 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.026568    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f27715f8-628e-434c-8ef8-40f2a61c5ad4-proxy-tls\") pod \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") "
Sep 17 17:08:34.027024 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.026625    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-rbac-proxy-sar-config\") pod \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") "
Sep 17 17:08:34.027024 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.026712    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkqwk\" (UniqueName: \"kubernetes.io/projected/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-api-access-jkqwk\") pod \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") "
Sep 17 17:08:34.027024 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.026738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-cabundle-cert\") pod \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") "
Sep 17 17:08:34.027024 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.026786    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kserve-provision-location\") pod \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\" (UID: \"f27715f8-628e-434c-8ef8-40f2a61c5ad4\") "
Sep 17 17:08:34.027245 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.027036    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f27715f8-628e-434c-8ef8-40f2a61c5ad4" (UID: "f27715f8-628e-434c-8ef8-40f2a61c5ad4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:08:34.027245 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.027135    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f27715f8-628e-434c-8ef8-40f2a61c5ad4" (UID: "f27715f8-628e-434c-8ef8-40f2a61c5ad4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:08:34.028716 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.028689    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27715f8-628e-434c-8ef8-40f2a61c5ad4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f27715f8-628e-434c-8ef8-40f2a61c5ad4" (UID: "f27715f8-628e-434c-8ef8-40f2a61c5ad4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:08:34.029189 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.029163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-api-access-jkqwk" (OuterVolumeSpecName: "kube-api-access-jkqwk") pod "f27715f8-628e-434c-8ef8-40f2a61c5ad4" (UID: "f27715f8-628e-434c-8ef8-40f2a61c5ad4"). InnerVolumeSpecName "kube-api-access-jkqwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:08:34.054785 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.054761    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f27715f8-628e-434c-8ef8-40f2a61c5ad4" (UID: "f27715f8-628e-434c-8ef8-40f2a61c5ad4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:08:34.127929 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.127906    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkqwk\" (UniqueName: \"kubernetes.io/projected/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-api-access-jkqwk\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:08:34.127929 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.127928    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:08:34.128047 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.127938    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:08:34.128047 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.127946    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f27715f8-628e-434c-8ef8-40f2a61c5ad4-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:08:34.128047 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.127954    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f27715f8-628e-434c-8ef8-40f2a61c5ad4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:08:34.500414 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.500377    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 17:08:34.685830 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.685795    2566 generic.go:356] "Generic (PLEG): container finished" podID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerID="d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81" exitCode=0
Sep 17 17:08:34.685981 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.685858    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"
Sep 17 17:08:34.685981 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.685863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" event={"ID":"f27715f8-628e-434c-8ef8-40f2a61c5ad4","Type":"ContainerDied","Data":"d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81"}
Sep 17 17:08:34.685981 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.685930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk" event={"ID":"f27715f8-628e-434c-8ef8-40f2a61c5ad4","Type":"ContainerDied","Data":"50e0f8f1b162f6fdfcf7f31bcbc506b742962659c63f868499fe9d4adc7c2304"}
Sep 17 17:08:34.685981 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.685970    2566 scope.go:117] "RemoveContainer" containerID="a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04"
Sep 17 17:08:34.696098 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.696081    2566 scope.go:117] "RemoveContainer" containerID="d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81"
Sep 17 17:08:34.707019 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.706991    2566 scope.go:117] "RemoveContainer" containerID="6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f"
Sep 17 17:08:34.714680 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.714656    2566 scope.go:117] "RemoveContainer" containerID="a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04"
Sep 17 17:08:34.715223 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:08:34.715202    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04\": container with ID starting with a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04 not found: ID does not exist" containerID="a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04"
Sep 17 17:08:34.715550 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.715228    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04"} err="failed to get container status \"a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04\": rpc error: code = NotFound desc = could not find container \"a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04\": container with ID starting with a35b3da4c0949de0d7cb58fe1ddfc1b8553997ebea5367f6615939dbfe0f9e04 not found: ID does not exist"
Sep 17 17:08:34.715550 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.715243    2566 scope.go:117] "RemoveContainer" containerID="d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81"
Sep 17 17:08:34.715550 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:08:34.715510    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81\": container with ID starting with d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81 not found: ID does not exist" containerID="d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81"
Sep 17 17:08:34.715550 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.715538    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81"} err="failed to get container status \"d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81\": rpc error: code = NotFound desc = could not find container \"d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81\": container with ID starting with d7c9cd337571c1fced0e6f6f3cd605d2ccff28cd2ac9029b7ea1839061913d81 not found: ID does not exist"
Sep 17 17:08:34.715550 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.715552    2566 scope.go:117] "RemoveContainer" containerID="6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f"
Sep 17 17:08:34.715833 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:08:34.715815    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f\": container with ID starting with 6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f not found: ID does not exist" containerID="6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f"
Sep 17 17:08:34.715876 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.715838    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f"} err="failed to get container status \"6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f\": rpc error: code = NotFound desc = could not find container \"6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f\": container with ID starting with 6137362c4cf713d156e82afca5b765d4aed89561f73370fa5d9542c3b9fb874f not found: ID does not exist"
Sep 17 17:08:34.716878 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.716861    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"]
Sep 17 17:08:34.732923 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:34.732902    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7fbff8696d-wttbk"]
Sep 17 17:08:35.499767 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:35.499731    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 17:08:35.809923 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:35.809848    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" path="/var/lib/kubelet/pods/f27715f8-628e-434c-8ef8-40f2a61c5ad4/volumes"
Sep 17 17:08:38.132823 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.132787    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:08:38.162953 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.162928    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"]
Sep 17 17:08:38.163310 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163297    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="kube-rbac-proxy"
Sep 17 17:08:38.163365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163312    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="kube-rbac-proxy"
Sep 17 17:08:38.163365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163322    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="storage-initializer"
Sep 17 17:08:38.163365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163328    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="storage-initializer"
Sep 17 17:08:38.163365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163334    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kserve-container"
Sep 17 17:08:38.163365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163340    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kserve-container"
Sep 17 17:08:38.163365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163351    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kube-rbac-proxy"
Sep 17 17:08:38.163365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163356    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kube-rbac-proxy"
Sep 17 17:08:38.163365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163361    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="storage-initializer"
Sep 17 17:08:38.163365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163367    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="storage-initializer"
Sep 17 17:08:38.163775 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163373    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="kserve-container"
Sep 17 17:08:38.163775 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163377    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="kserve-container"
Sep 17 17:08:38.163775 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163432    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kube-rbac-proxy"
Sep 17 17:08:38.163775 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163442    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d64c0c-90ea-4566-a081-a51bee1c8d3d" containerName="kserve-container"
Sep 17 17:08:38.163775 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163449    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="kube-rbac-proxy"
Sep 17 17:08:38.163775 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.163456    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27715f8-628e-434c-8ef8-40f2a61c5ad4" containerName="kserve-container"
Sep 17 17:08:38.166721 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.166696    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.169001 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.168977    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 17 17:08:38.169124 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.169107    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 17:08:38.177303 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.177282    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"]
Sep 17 17:08:38.256485 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.256448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.256636 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.256499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.256636 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.256517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.256636 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.256604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78bw\" (UniqueName: \"kubernetes.io/projected/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-api-access-l78bw\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.256800 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.256653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.357934 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.357896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.358123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.357953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.358123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.357979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.358123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.358025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78bw\" (UniqueName: \"kubernetes.io/projected/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-api-access-l78bw\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.358123 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:08:38.358058    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 17:08:38.358123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.358061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.358434 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:08:38.358139    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-proxy-tls podName:ba0bede4-f20c-4ca5-82c8-91ae6c689fb3 nodeName:}" failed. No retries permitted until 2026-09-17 17:08:38.85811727 +0000 UTC m=+1627.570124530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" (UID: "ba0bede4-f20c-4ca5-82c8-91ae6c689fb3") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 17:08:38.358499 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.358456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.358722 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.358699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.358798 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.358707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.366321 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.366300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78bw\" (UniqueName: \"kubernetes.io/projected/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-api-access-l78bw\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.689488 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.689434    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:08:38.863549 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.863513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:38.865890 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:38.865872    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:39.078276 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:39.078234    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:39.353511 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:39.353490    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"]
Sep 17 17:08:39.354945 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:08:39.354914    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0bede4_f20c_4ca5_82c8_91ae6c689fb3.slice/crio-3fbe3ffbbb5df37e5f690bfa524307a0973cf0321794d5e3f9fe09821404ea87 WatchSource:0}: Error finding container 3fbe3ffbbb5df37e5f690bfa524307a0973cf0321794d5e3f9fe09821404ea87: Status 404 returned error can't find the container with id 3fbe3ffbbb5df37e5f690bfa524307a0973cf0321794d5e3f9fe09821404ea87
Sep 17 17:08:39.713058 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:39.712961    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" event={"ID":"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3","Type":"ContainerStarted","Data":"3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5"}
Sep 17 17:08:39.713058 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:39.713024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" event={"ID":"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3","Type":"ContainerStarted","Data":"3fbe3ffbbb5df37e5f690bfa524307a0973cf0321794d5e3f9fe09821404ea87"}
Sep 17 17:08:43.733713 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:43.733679    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerID="3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5" exitCode=0
Sep 17 17:08:43.734232 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:43.733719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" event={"ID":"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3","Type":"ContainerDied","Data":"3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5"}
Sep 17 17:08:44.501112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:44.501074    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 17:08:44.741133 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:44.741094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" event={"ID":"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3","Type":"ContainerStarted","Data":"9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59"}
Sep 17 17:08:44.741571 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:44.741159    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" event={"ID":"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3","Type":"ContainerStarted","Data":"f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9"}
Sep 17 17:08:44.741571 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:44.741378    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:44.741571 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:44.741425    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:44.761405 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:44.761299    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" podStartSLOduration=2.954900415 podStartE2EDuration="6.761282841s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.806382426s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:08:44.760967061 +0000 UTC m=+1633.472974330" watchObservedRunningTime="2026-09-17 17:08:44.761282841 +0000 UTC m=+1633.473290110"
Sep 17 17:08:45.499795 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:45.499750    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 17:08:50.751629 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:50.751599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:08:54.500999 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:54.500956    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 17:08:55.500702 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:08:55.500665    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 17:09:04.088056 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:04.088024    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:04.500544 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:04.500504    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 17:09:05.499818 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:05.499778    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 17:09:14.501435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:14.501404    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:09:14.641404 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:14.641370    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"]
Sep 17 17:09:14.641634 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:14.641605    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container" containerID="cri-o://89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8" gracePeriod=30
Sep 17 17:09:14.641720 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:14.641659    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kube-rbac-proxy" containerID="cri-o://75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5" gracePeriod=30
Sep 17 17:09:14.641875 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:14.641851    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 17:09:14.895684 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:14.895649    2566 generic.go:356] "Generic (PLEG): container finished" podID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerID="75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5" exitCode=2
Sep 17 17:09:14.895833 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:14.895720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" event={"ID":"dda84025-b5a8-4825-bf31-2ec23ccd695b","Type":"ContainerDied","Data":"75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5"}
Sep 17 17:09:15.495317 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:15.495274    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 17 17:09:19.284220 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.284196    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:09:19.405051 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.404964    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda84025-b5a8-4825-bf31-2ec23ccd695b-proxy-tls\") pod \"dda84025-b5a8-4825-bf31-2ec23ccd695b\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") "
Sep 17 17:09:19.405051 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.405025    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dda84025-b5a8-4825-bf31-2ec23ccd695b-kserve-provision-location\") pod \"dda84025-b5a8-4825-bf31-2ec23ccd695b\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") "
Sep 17 17:09:19.405314 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.405069    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-cabundle-cert\") pod \"dda84025-b5a8-4825-bf31-2ec23ccd695b\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") "
Sep 17 17:09:19.405314 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.405091    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-rbac-proxy-sar-config\") pod \"dda84025-b5a8-4825-bf31-2ec23ccd695b\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") "
Sep 17 17:09:19.405314 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.405123    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldk29\" (UniqueName: \"kubernetes.io/projected/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-api-access-ldk29\") pod \"dda84025-b5a8-4825-bf31-2ec23ccd695b\" (UID: \"dda84025-b5a8-4825-bf31-2ec23ccd695b\") "
Sep 17 17:09:19.405487 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.405420    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda84025-b5a8-4825-bf31-2ec23ccd695b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dda84025-b5a8-4825-bf31-2ec23ccd695b" (UID: "dda84025-b5a8-4825-bf31-2ec23ccd695b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:19.405555 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.405473    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dda84025-b5a8-4825-bf31-2ec23ccd695b" (UID: "dda84025-b5a8-4825-bf31-2ec23ccd695b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:19.405555 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.405485    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dda84025-b5a8-4825-bf31-2ec23ccd695b" (UID: "dda84025-b5a8-4825-bf31-2ec23ccd695b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:19.407232 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.407205    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-api-access-ldk29" (OuterVolumeSpecName: "kube-api-access-ldk29") pod "dda84025-b5a8-4825-bf31-2ec23ccd695b" (UID: "dda84025-b5a8-4825-bf31-2ec23ccd695b"). InnerVolumeSpecName "kube-api-access-ldk29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:09:19.407319 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.407239    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda84025-b5a8-4825-bf31-2ec23ccd695b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dda84025-b5a8-4825-bf31-2ec23ccd695b" (UID: "dda84025-b5a8-4825-bf31-2ec23ccd695b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:09:19.506450 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.506425    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dda84025-b5a8-4825-bf31-2ec23ccd695b-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:19.506450 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.506453    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:19.506617 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.506467    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:19.506617 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.506479    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldk29\" (UniqueName: \"kubernetes.io/projected/dda84025-b5a8-4825-bf31-2ec23ccd695b-kube-api-access-ldk29\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:19.506617 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.506491    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda84025-b5a8-4825-bf31-2ec23ccd695b-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:19.920848 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.920818    2566 generic.go:356] "Generic (PLEG): container finished" podID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerID="89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8" exitCode=139
Sep 17 17:09:19.920992 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.920890    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"
Sep 17 17:09:19.920992 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.920891    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" event={"ID":"dda84025-b5a8-4825-bf31-2ec23ccd695b","Type":"ContainerDied","Data":"89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8"}
Sep 17 17:09:19.921113 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.921000    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g" event={"ID":"dda84025-b5a8-4825-bf31-2ec23ccd695b","Type":"ContainerDied","Data":"20bb0cbaadac2431755a620cf1400927378a0567726e4624dce6d633ff619362"}
Sep 17 17:09:19.921113 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.921034    2566 scope.go:117] "RemoveContainer" containerID="75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5"
Sep 17 17:09:19.929484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.929467    2566 scope.go:117] "RemoveContainer" containerID="89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8"
Sep 17 17:09:19.937237 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.937218    2566 scope.go:117] "RemoveContainer" containerID="bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b"
Sep 17 17:09:19.938223 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.938209    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"]
Sep 17 17:09:19.941992 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.941972    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-r462g"]
Sep 17 17:09:19.945108 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.945092    2566 scope.go:117] "RemoveContainer" containerID="75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5"
Sep 17 17:09:19.945359 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:09:19.945339    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5\": container with ID starting with 75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5 not found: ID does not exist" containerID="75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5"
Sep 17 17:09:19.945424 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.945367    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5"} err="failed to get container status \"75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5\": rpc error: code = NotFound desc = could not find container \"75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5\": container with ID starting with 75c84e241b10b212b78f450fa023c6bc0a1cb0d70b46096a8acc85b3e6f3a8e5 not found: ID does not exist"
Sep 17 17:09:19.945424 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.945382    2566 scope.go:117] "RemoveContainer" containerID="89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8"
Sep 17 17:09:19.945602 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:09:19.945583    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8\": container with ID starting with 89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8 not found: ID does not exist" containerID="89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8"
Sep 17 17:09:19.945641 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.945607    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8"} err="failed to get container status \"89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8\": rpc error: code = NotFound desc = could not find container \"89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8\": container with ID starting with 89b19d43601d0f4c0839e395e6c8431758d3563ecb38817c75444f56c59a6cb8 not found: ID does not exist"
Sep 17 17:09:19.945641 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.945620    2566 scope.go:117] "RemoveContainer" containerID="bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b"
Sep 17 17:09:19.945861 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:09:19.945845    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b\": container with ID starting with bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b not found: ID does not exist" containerID="bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b"
Sep 17 17:09:19.945906 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:19.945866    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b"} err="failed to get container status \"bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b\": rpc error: code = NotFound desc = could not find container \"bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b\": container with ID starting with bb19d21cadd2fdc3d89a113ffcb4cfb5cfe88b3ca73007d997e1e592f2208e3b not found: ID does not exist"
Sep 17 17:09:20.825453 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:20.825408    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 17:09:21.809787 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:21.809754    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" path="/var/lib/kubelet/pods/dda84025-b5a8-4825-bf31-2ec23ccd695b/volumes"
Sep 17 17:09:30.754229 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:30.754198    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:09:32.988720 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:32.988688    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"]
Sep 17 17:09:32.989557 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:32.989495    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container" containerID="cri-o://5e9bf5d269b7399953a286ec904f2f4be13e8a444851a66891cff1524df940df" gracePeriod=30
Sep 17 17:09:32.989697 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:32.989521    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kube-rbac-proxy" containerID="cri-o://be4b8fca850b127eedc12854a0889827c89181416c47772f15e3682f2e559197" gracePeriod=30
Sep 17 17:09:33.012498 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:33.012471    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:33.063471 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:33.063441    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:33.996701 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:33.996673    2566 generic.go:356] "Generic (PLEG): container finished" podID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerID="be4b8fca850b127eedc12854a0889827c89181416c47772f15e3682f2e559197" exitCode=2
Sep 17 17:09:33.997067 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:33.996748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" event={"ID":"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba","Type":"ContainerDied","Data":"be4b8fca850b127eedc12854a0889827c89181416c47772f15e3682f2e559197"}
Sep 17 17:09:34.490622 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:34.490589    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 17 17:09:34.500755 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:34.500729    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 17:09:37.014289 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.014224    2566 generic.go:356] "Generic (PLEG): container finished" podID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerID="5e9bf5d269b7399953a286ec904f2f4be13e8a444851a66891cff1524df940df" exitCode=0
Sep 17 17:09:37.014596 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.014292    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" event={"ID":"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba","Type":"ContainerDied","Data":"5e9bf5d269b7399953a286ec904f2f4be13e8a444851a66891cff1524df940df"}
Sep 17 17:09:37.041112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.041092    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:09:37.149842 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.149772    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67gfx\" (UniqueName: \"kubernetes.io/projected/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-api-access-67gfx\") pod \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") "
Sep 17 17:09:37.149842 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.149834    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-rbac-proxy-sar-config\") pod \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") "
Sep 17 17:09:37.149997 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.149860    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-cabundle-cert\") pod \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") "
Sep 17 17:09:37.149997 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.149918    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-proxy-tls\") pod \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") "
Sep 17 17:09:37.149997 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.149950    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kserve-provision-location\") pod \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\" (UID: \"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba\") "
Sep 17 17:09:37.150327 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.150301    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" (UID: "6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:37.150415 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.150347    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" (UID: "6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:37.150415 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.150355    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" (UID: "6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:37.151950 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.151922    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-api-access-67gfx" (OuterVolumeSpecName: "kube-api-access-67gfx") pod "6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" (UID: "6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba"). InnerVolumeSpecName "kube-api-access-67gfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:09:37.151950 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.151932    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" (UID: "6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:09:37.251123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.251098    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67gfx\" (UniqueName: \"kubernetes.io/projected/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-api-access-67gfx\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:37.251123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.251119    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:37.251274 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.251146    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:37.251274 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.251157    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:37.251274 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:37.251164    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:38.020651 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.020622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988" event={"ID":"6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba","Type":"ContainerDied","Data":"31c1cadd4d79ec97c597a0cad2ab0759f2e88c753c4165f2d59f91e0640bcca8"}
Sep 17 17:09:38.021015 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.020687    2566 scope.go:117] "RemoveContainer" containerID="be4b8fca850b127eedc12854a0889827c89181416c47772f15e3682f2e559197"
Sep 17 17:09:38.021015 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.020635    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"
Sep 17 17:09:38.031375 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.031329    2566 scope.go:117] "RemoveContainer" containerID="5e9bf5d269b7399953a286ec904f2f4be13e8a444851a66891cff1524df940df"
Sep 17 17:09:38.039289 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.039268    2566 scope.go:117] "RemoveContainer" containerID="6764dac8686974e27cc87d7096d2f0e92de06e62b5075db371b4a0e5cb88e805"
Sep 17 17:09:38.046003 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.045980    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"]
Sep 17 17:09:38.048914 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.048892    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-b7988"]
Sep 17 17:09:38.163702 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.163619    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:38.249466 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.249418    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:38.272632 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.272603    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"]
Sep 17 17:09:38.272984 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.272971    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container"
Sep 17 17:09:38.273026 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.272986    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container"
Sep 17 17:09:38.273026 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273000    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kube-rbac-proxy"
Sep 17 17:09:38.273026 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273005    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kube-rbac-proxy"
Sep 17 17:09:38.273026 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273018    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="storage-initializer"
Sep 17 17:09:38.273026 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273024    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="storage-initializer"
Sep 17 17:09:38.273188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273030    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kube-rbac-proxy"
Sep 17 17:09:38.273188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273036    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kube-rbac-proxy"
Sep 17 17:09:38.273188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273050    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="storage-initializer"
Sep 17 17:09:38.273188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273056    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="storage-initializer"
Sep 17 17:09:38.273188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273068    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container"
Sep 17 17:09:38.273188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273073    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container"
Sep 17 17:09:38.273188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273132    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kserve-container"
Sep 17 17:09:38.273188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273142    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kserve-container"
Sep 17 17:09:38.273188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273149    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dda84025-b5a8-4825-bf31-2ec23ccd695b" containerName="kube-rbac-proxy"
Sep 17 17:09:38.273188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.273156    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" containerName="kube-rbac-proxy"
Sep 17 17:09:38.275943 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.275926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.278510 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.278479    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 17:09:38.278614 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.278534    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 17:09:38.287365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.287347    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"]
Sep 17 17:09:38.358021 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.357991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.358199 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.358025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.358199 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.358101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa20d16-ee5d-40c9-88de-cc8969f2da58-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.358199 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.358140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8gzd\" (UniqueName: \"kubernetes.io/projected/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-api-access-q8gzd\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.358199 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.358170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.388939 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.388912    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"]
Sep 17 17:09:38.389154 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.389124    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="kserve-container" containerID="cri-o://f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9" gracePeriod=30
Sep 17 17:09:38.389270 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.389155    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="kube-rbac-proxy" containerID="cri-o://9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59" gracePeriod=30
Sep 17 17:09:38.425811 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.425740    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:38.459559 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.459527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa20d16-ee5d-40c9-88de-cc8969f2da58-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.459659 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.459566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8gzd\" (UniqueName: \"kubernetes.io/projected/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-api-access-q8gzd\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.459659 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.459592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.459659 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.459627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.459659 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.459644    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.460140 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.460116    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.460322 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.460304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.460381 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.460357    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.461945 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.461923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa20d16-ee5d-40c9-88de-cc8969f2da58-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.467567 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.467542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8gzd\" (UniqueName: \"kubernetes.io/projected/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-api-access-q8gzd\") pod \"isvc-canary-promote-predictor-8bc5b857-45fh4\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.524267 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.524230    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:38.586872 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.586850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:38.622725 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.622685    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:38.858073 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:38.858052    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"]
Sep 17 17:09:38.860445 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:09:38.860416    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa20d16_ee5d_40c9_88de_cc8969f2da58.slice/crio-381a90bbc0bc4b901802e306a8babfc01141701bab99bbdcaa36aaec3ee6992d WatchSource:0}: Error finding container 381a90bbc0bc4b901802e306a8babfc01141701bab99bbdcaa36aaec3ee6992d: Status 404 returned error can't find the container with id 381a90bbc0bc4b901802e306a8babfc01141701bab99bbdcaa36aaec3ee6992d
Sep 17 17:09:39.029389 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:39.029292    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" event={"ID":"8aa20d16-ee5d-40c9-88de-cc8969f2da58","Type":"ContainerStarted","Data":"b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067"}
Sep 17 17:09:39.029851 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:39.029409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" event={"ID":"8aa20d16-ee5d-40c9-88de-cc8969f2da58","Type":"ContainerStarted","Data":"381a90bbc0bc4b901802e306a8babfc01141701bab99bbdcaa36aaec3ee6992d"}
Sep 17 17:09:39.031887 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:39.031855    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerID="9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59" exitCode=2
Sep 17 17:09:39.032027 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:39.031910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" event={"ID":"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3","Type":"ContainerDied","Data":"9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59"}
Sep 17 17:09:39.810874 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:39.810839    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba" path="/var/lib/kubelet/pods/6c6e48ad-d5e5-44ba-bbe2-2b07f04aabba/volumes"
Sep 17 17:09:40.746730 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:40.746696    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Sep 17 17:09:43.052238 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:43.052204    2566 generic.go:356] "Generic (PLEG): container finished" podID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerID="b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067" exitCode=0
Sep 17 17:09:43.052640 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:43.052293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" event={"ID":"8aa20d16-ee5d-40c9-88de-cc8969f2da58","Type":"ContainerDied","Data":"b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067"}
Sep 17 17:09:43.352128 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:43.352047    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:43.760890 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:43.760855    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:44.059782 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:44.059708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" event={"ID":"8aa20d16-ee5d-40c9-88de-cc8969f2da58","Type":"ContainerStarted","Data":"14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852"}
Sep 17 17:09:44.059782 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:44.059771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" event={"ID":"8aa20d16-ee5d-40c9-88de-cc8969f2da58","Type":"ContainerStarted","Data":"2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54"}
Sep 17 17:09:44.060173 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:44.060045    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:44.060173 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:44.060159    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:44.061496 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:44.061474    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 17:09:44.080316 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:44.080271    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podStartSLOduration=2.359150779 podStartE2EDuration="6.080247531s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.721096752s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:09:44.078113866 +0000 UTC m=+1692.790121135" watchObservedRunningTime="2026-09-17 17:09:44.080247531 +0000 UTC m=+1692.792254798"
Sep 17 17:09:44.847272 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:44.847234    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:09:45.015188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.015165    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-proxy-tls\") pod \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") "
Sep 17 17:09:45.015346 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.015201    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-rbac-proxy-sar-config\") pod \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") "
Sep 17 17:09:45.015346 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.015279    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kserve-provision-location\") pod \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") "
Sep 17 17:09:45.015346 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.015311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-cabundle-cert\") pod \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") "
Sep 17 17:09:45.015495 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.015359    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l78bw\" (UniqueName: \"kubernetes.io/projected/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-api-access-l78bw\") pod \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\" (UID: \"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3\") "
Sep 17 17:09:45.015614 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.015589    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" (UID: "ba0bede4-f20c-4ca5-82c8-91ae6c689fb3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:45.015708 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.015633    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" (UID: "ba0bede4-f20c-4ca5-82c8-91ae6c689fb3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:45.015761 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.015703    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" (UID: "ba0bede4-f20c-4ca5-82c8-91ae6c689fb3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:45.017220 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.017202    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" (UID: "ba0bede4-f20c-4ca5-82c8-91ae6c689fb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:09:45.017514 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.017492    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-api-access-l78bw" (OuterVolumeSpecName: "kube-api-access-l78bw") pod "ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" (UID: "ba0bede4-f20c-4ca5-82c8-91ae6c689fb3"). InnerVolumeSpecName "kube-api-access-l78bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:09:45.067042 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.067009    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerID="f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9" exitCode=0
Sep 17 17:09:45.067436 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.067061    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"
Sep 17 17:09:45.067436 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.067096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" event={"ID":"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3","Type":"ContainerDied","Data":"f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9"}
Sep 17 17:09:45.067436 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.067211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt" event={"ID":"ba0bede4-f20c-4ca5-82c8-91ae6c689fb3","Type":"ContainerDied","Data":"3fbe3ffbbb5df37e5f690bfa524307a0973cf0321794d5e3f9fe09821404ea87"}
Sep 17 17:09:45.067436 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.067270    2566 scope.go:117] "RemoveContainer" containerID="9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59"
Sep 17 17:09:45.067686 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.067661    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 17:09:45.076574 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.076558    2566 scope.go:117] "RemoveContainer" containerID="f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9"
Sep 17 17:09:45.084527 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.084509    2566 scope.go:117] "RemoveContainer" containerID="3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5"
Sep 17 17:09:45.090025 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.090001    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"]
Sep 17 17:09:45.092128 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.092116    2566 scope.go:117] "RemoveContainer" containerID="9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59"
Sep 17 17:09:45.092396 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:09:45.092378    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59\": container with ID starting with 9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59 not found: ID does not exist" containerID="9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59"
Sep 17 17:09:45.092463 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.092406    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59"} err="failed to get container status \"9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59\": rpc error: code = NotFound desc = could not find container \"9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59\": container with ID starting with 9af878ed850675a3869772c6eef4fd331b044e560927f7de3bc3133687e38f59 not found: ID does not exist"
Sep 17 17:09:45.092463 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.092428    2566 scope.go:117] "RemoveContainer" containerID="f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9"
Sep 17 17:09:45.092688 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:09:45.092653    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9\": container with ID starting with f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9 not found: ID does not exist" containerID="f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9"
Sep 17 17:09:45.092727 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.092699    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9"} err="failed to get container status \"f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9\": rpc error: code = NotFound desc = could not find container \"f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9\": container with ID starting with f7f5e3a4335290b6d6bb4b98d78cba3eb2195668f538541278bc84aa242b0bb9 not found: ID does not exist"
Sep 17 17:09:45.092727 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.092720    2566 scope.go:117] "RemoveContainer" containerID="3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5"
Sep 17 17:09:45.092975 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:09:45.092961    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5\": container with ID starting with 3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5 not found: ID does not exist" containerID="3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5"
Sep 17 17:09:45.093030 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.092978    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5"} err="failed to get container status \"3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5\": rpc error: code = NotFound desc = could not find container \"3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5\": container with ID starting with 3d39ff2b25ec3564878136439dabae113e465de5d84b20ca44ccac072f2a2ed5 not found: ID does not exist"
Sep 17 17:09:45.095984 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.095965    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bhkvt"]
Sep 17 17:09:45.116407 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.116383    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:45.116407 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.116406    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:45.116528 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.116421    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l78bw\" (UniqueName: \"kubernetes.io/projected/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-api-access-l78bw\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:45.116528 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.116432    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:45.116528 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.116441    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:09:45.809627 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:45.809591    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" path="/var/lib/kubelet/pods/ba0bede4-f20c-4ca5-82c8-91ae6c689fb3/volumes"
Sep 17 17:09:50.072561 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:50.072528    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:09:50.073100 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:09:50.073071    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 17:10:00.073943 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:10:00.073851    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 17:10:10.073061 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:10:10.073020    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 17:10:20.073411 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:10:20.073372    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 17:10:30.073488 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:10:30.073450    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 17:10:40.073419 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:10:40.073381    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 17:10:50.073534 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:10:50.073496    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 17:10:57.809570 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:10:57.809543    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:11:08.236044 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.236000    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"]
Sep 17 17:11:08.236627 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.236332    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container" containerID="cri-o://2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54" gracePeriod=30
Sep 17 17:11:08.236627 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.236394    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kube-rbac-proxy" containerID="cri-o://14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852" gracePeriod=30
Sep 17 17:11:08.266158 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.266121    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:11:08.294538 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.294515    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"]
Sep 17 17:11:08.294861 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.294843    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="kube-rbac-proxy"
Sep 17 17:11:08.294903 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.294864    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="kube-rbac-proxy"
Sep 17 17:11:08.294903 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.294883    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="storage-initializer"
Sep 17 17:11:08.294903 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.294889    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="storage-initializer"
Sep 17 17:11:08.294903 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.294896    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="kserve-container"
Sep 17 17:11:08.294903 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.294902    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="kserve-container"
Sep 17 17:11:08.295061 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.294960    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="kserve-container"
Sep 17 17:11:08.295061 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.294969    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba0bede4-f20c-4ca5-82c8-91ae6c689fb3" containerName="kube-rbac-proxy"
Sep 17 17:11:08.298099 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.298083    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.300510 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.300491    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 17:11:08.307809 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.307787    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"]
Sep 17 17:11:08.450198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.450166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.450354 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.450221    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.450354 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.450311    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2fxd\" (UniqueName: \"kubernetes.io/projected/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-api-access-f2fxd\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.450354 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.450350    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.450511 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.450399    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.469548 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.469522    2566 generic.go:356] "Generic (PLEG): container finished" podID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerID="14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852" exitCode=2
Sep 17 17:11:08.469644 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.469582    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" event={"ID":"8aa20d16-ee5d-40c9-88de-cc8969f2da58","Type":"ContainerDied","Data":"14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852"}
Sep 17 17:11:08.551166 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.551090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.551166 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.551147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2fxd\" (UniqueName: \"kubernetes.io/projected/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-api-access-f2fxd\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.551379 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.551188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.551379 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.551237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.551379 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.551328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.551728 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.551702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.551840 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.551822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.551909 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.551891    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.553724 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.553699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.559060 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.559026    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2fxd\" (UniqueName: \"kubernetes.io/projected/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-api-access-f2fxd\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-fkh9w\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.609153 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.609130    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:08.881153 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:08.881131    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"]
Sep 17 17:11:08.883044 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:11:08.883016    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd6115a_30d8_4e68_9c45_1c1ce4b4dcb2.slice/crio-b6aaf1b4024b3b879b763ecd32c5dea004d5392cbd9f5a2768da227a248d9c23 WatchSource:0}: Error finding container b6aaf1b4024b3b879b763ecd32c5dea004d5392cbd9f5a2768da227a248d9c23: Status 404 returned error can't find the container with id b6aaf1b4024b3b879b763ecd32c5dea004d5392cbd9f5a2768da227a248d9c23
Sep 17 17:11:09.476331 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:09.476292    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" event={"ID":"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2","Type":"ContainerStarted","Data":"2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0"}
Sep 17 17:11:09.476687 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:09.476353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" event={"ID":"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2","Type":"ContainerStarted","Data":"b6aaf1b4024b3b879b763ecd32c5dea004d5392cbd9f5a2768da227a248d9c23"}
Sep 17 17:11:10.068500 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:10.068459    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 17 17:11:12.277601 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.277573    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:11:12.387347 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.387287    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa20d16-ee5d-40c9-88de-cc8969f2da58-proxy-tls\") pod \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") "
Sep 17 17:11:12.387347 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.387325    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-rbac-proxy-sar-config\") pod \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") "
Sep 17 17:11:12.387529 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.387350    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-cabundle-cert\") pod \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") "
Sep 17 17:11:12.387529 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.387380    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kserve-provision-location\") pod \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") "
Sep 17 17:11:12.387529 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.387441    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8gzd\" (UniqueName: \"kubernetes.io/projected/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-api-access-q8gzd\") pod \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\" (UID: \"8aa20d16-ee5d-40c9-88de-cc8969f2da58\") "
Sep 17 17:11:12.387734 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.387705    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8aa20d16-ee5d-40c9-88de-cc8969f2da58" (UID: "8aa20d16-ee5d-40c9-88de-cc8969f2da58"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:11:12.387796 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.387709    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8aa20d16-ee5d-40c9-88de-cc8969f2da58" (UID: "8aa20d16-ee5d-40c9-88de-cc8969f2da58"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:11:12.387796 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.387746    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8aa20d16-ee5d-40c9-88de-cc8969f2da58" (UID: "8aa20d16-ee5d-40c9-88de-cc8969f2da58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:12.389575 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.389550    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa20d16-ee5d-40c9-88de-cc8969f2da58-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8aa20d16-ee5d-40c9-88de-cc8969f2da58" (UID: "8aa20d16-ee5d-40c9-88de-cc8969f2da58"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:11:12.389853 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.389824    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-api-access-q8gzd" (OuterVolumeSpecName: "kube-api-access-q8gzd") pod "8aa20d16-ee5d-40c9-88de-cc8969f2da58" (UID: "8aa20d16-ee5d-40c9-88de-cc8969f2da58"). InnerVolumeSpecName "kube-api-access-q8gzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:11:12.488770 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.488750    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa20d16-ee5d-40c9-88de-cc8969f2da58-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:11:12.488770 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.488768    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:11:12.488904 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.488777    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa20d16-ee5d-40c9-88de-cc8969f2da58-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:11:12.488904 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.488786    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:11:12.488904 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.488794    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8gzd\" (UniqueName: \"kubernetes.io/projected/8aa20d16-ee5d-40c9-88de-cc8969f2da58-kube-api-access-q8gzd\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:11:12.493491 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.493461    2566 generic.go:356] "Generic (PLEG): container finished" podID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerID="2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54" exitCode=0
Sep 17 17:11:12.493603 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.493501    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" event={"ID":"8aa20d16-ee5d-40c9-88de-cc8969f2da58","Type":"ContainerDied","Data":"2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54"}
Sep 17 17:11:12.493603 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.493515    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"
Sep 17 17:11:12.493603 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.493564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4" event={"ID":"8aa20d16-ee5d-40c9-88de-cc8969f2da58","Type":"ContainerDied","Data":"381a90bbc0bc4b901802e306a8babfc01141701bab99bbdcaa36aaec3ee6992d"}
Sep 17 17:11:12.493603 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.493602    2566 scope.go:117] "RemoveContainer" containerID="14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852"
Sep 17 17:11:12.502337 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.502318    2566 scope.go:117] "RemoveContainer" containerID="2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54"
Sep 17 17:11:12.509789 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.509771    2566 scope.go:117] "RemoveContainer" containerID="b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067"
Sep 17 17:11:12.515829 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.515807    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"]
Sep 17 17:11:12.518061 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.518044    2566 scope.go:117] "RemoveContainer" containerID="14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852"
Sep 17 17:11:12.518424 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:11:12.518402    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852\": container with ID starting with 14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852 not found: ID does not exist" containerID="14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852"
Sep 17 17:11:12.518513 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.518434    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852"} err="failed to get container status \"14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852\": rpc error: code = NotFound desc = could not find container \"14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852\": container with ID starting with 14e5aa1db081744f50940ba6a3b85fc73de86fa5c05c42997a58d3622b643852 not found: ID does not exist"
Sep 17 17:11:12.518513 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.518486    2566 scope.go:117] "RemoveContainer" containerID="2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54"
Sep 17 17:11:12.518777 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:11:12.518759    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54\": container with ID starting with 2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54 not found: ID does not exist" containerID="2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54"
Sep 17 17:11:12.518856 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.518785    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54"} err="failed to get container status \"2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54\": rpc error: code = NotFound desc = could not find container \"2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54\": container with ID starting with 2dc4bb65257e1d769f2ab2728559228704e718c64ef5bcadc6fced491d459b54 not found: ID does not exist"
Sep 17 17:11:12.518856 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.518800    2566 scope.go:117] "RemoveContainer" containerID="b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067"
Sep 17 17:11:12.519066 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:11:12.519047    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067\": container with ID starting with b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067 not found: ID does not exist" containerID="b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067"
Sep 17 17:11:12.519124 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.519072    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067"} err="failed to get container status \"b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067\": rpc error: code = NotFound desc = could not find container \"b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067\": container with ID starting with b2cf59c5690a519286d12e21f9cc2652710386abdcc623bee43c291f24364067 not found: ID does not exist"
Sep 17 17:11:12.519684 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:12.519667    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-45fh4"]
Sep 17 17:11:13.502907 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:13.502871    2566 generic.go:356] "Generic (PLEG): container finished" podID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerID="2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0" exitCode=0
Sep 17 17:11:13.503377 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:13.502942    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" event={"ID":"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2","Type":"ContainerDied","Data":"2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0"}
Sep 17 17:11:13.537074 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:13.537027    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:11:13.810807 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:13.810735    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" path="/var/lib/kubelet/pods/8aa20d16-ee5d-40c9-88de-cc8969f2da58/volumes"
Sep 17 17:11:14.512502 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:14.512470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" event={"ID":"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2","Type":"ContainerStarted","Data":"f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb"}
Sep 17 17:11:14.512869 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:14.512526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" event={"ID":"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2","Type":"ContainerStarted","Data":"15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515"}
Sep 17 17:11:14.512869 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:14.512710    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:14.531886 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:14.531831    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podStartSLOduration=2.65244559 podStartE2EDuration="6.531811331s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.879365741s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:11:14.529099492 +0000 UTC m=+1783.241106786" watchObservedRunningTime="2026-09-17 17:11:14.531811331 +0000 UTC m=+1783.243818600"
Sep 17 17:11:15.517561 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:15.517532    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:15.518705 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:15.518683    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 17:11:16.521963 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:16.521923    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 17:11:18.659856 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:18.659817    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:11:18.848613 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:18.848572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:11:21.528096 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:21.528067    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:11:21.528608 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:21.528579    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 17:11:31.528565 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:31.528518    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 17:11:41.528895 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:41.528853    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 17:11:51.528524 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:11:51.528478    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 17:12:01.528955 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:01.528908    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 17:12:11.529025 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:11.528982    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 17:12:21.529248 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:21.529200    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 17:12:31.529413 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:31.529381    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:12:38.448380 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:38.448346    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"]
Sep 17 17:12:38.448847 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:38.448596    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container" containerID="cri-o://15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515" gracePeriod=30
Sep 17 17:12:38.448847 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:38.448635    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kube-rbac-proxy" containerID="cri-o://f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb" gracePeriod=30
Sep 17 17:12:38.472551 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:38.472521    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:12:38.906187 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:38.906152    2566 generic.go:356] "Generic (PLEG): container finished" podID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerID="f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb" exitCode=2
Sep 17 17:12:38.906380 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:38.906229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" event={"ID":"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2","Type":"ContainerDied","Data":"f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb"}
Sep 17 17:12:41.523113 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:41.523066    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 17 17:12:41.528679 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:41.528644    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 17:12:42.686219 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.686196    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:12:42.780123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.780062    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-cabundle-cert\") pod \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") "
Sep 17 17:12:42.780123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.780112    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kserve-provision-location\") pod \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") "
Sep 17 17:12:42.780319 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.780130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-proxy-tls\") pod \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") "
Sep 17 17:12:42.780319 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.780151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2fxd\" (UniqueName: \"kubernetes.io/projected/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-api-access-f2fxd\") pod \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") "
Sep 17 17:12:42.780319 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.780268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-rbac-proxy-sar-config\") pod \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\" (UID: \"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2\") "
Sep 17 17:12:42.780473 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.780459    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" (UID: "9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:12:42.780534 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.780448    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" (UID: "9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:12:42.780590 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.780541    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:12:42.780590 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.780559    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:12:42.780705 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.780683    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" (UID: "9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:12:42.782237 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.782211    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" (UID: "9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:12:42.782371 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.782294    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-api-access-f2fxd" (OuterVolumeSpecName: "kube-api-access-f2fxd") pod "9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" (UID: "9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2"). InnerVolumeSpecName "kube-api-access-f2fxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:12:42.881575 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.881549    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:12:42.881575 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.881571    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2fxd\" (UniqueName: \"kubernetes.io/projected/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-api-access-f2fxd\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:12:42.881575 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.881580    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:12:42.925068 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.925037    2566 generic.go:356] "Generic (PLEG): container finished" podID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerID="15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515" exitCode=0
Sep 17 17:12:42.925204 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.925087    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" event={"ID":"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2","Type":"ContainerDied","Data":"15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515"}
Sep 17 17:12:42.925204 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.925151    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w" event={"ID":"9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2","Type":"ContainerDied","Data":"b6aaf1b4024b3b879b763ecd32c5dea004d5392cbd9f5a2768da227a248d9c23"}
Sep 17 17:12:42.925204 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.925154    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"
Sep 17 17:12:42.925204 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.925187    2566 scope.go:117] "RemoveContainer" containerID="f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb"
Sep 17 17:12:42.936270 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.935901    2566 scope.go:117] "RemoveContainer" containerID="15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515"
Sep 17 17:12:42.944035 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.944019    2566 scope.go:117] "RemoveContainer" containerID="2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0"
Sep 17 17:12:42.949892 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.949867    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"]
Sep 17 17:12:42.952816 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.952799    2566 scope.go:117] "RemoveContainer" containerID="f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb"
Sep 17 17:12:42.952958 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.952941    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-fkh9w"]
Sep 17 17:12:42.953092 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:12:42.953074    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb\": container with ID starting with f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb not found: ID does not exist" containerID="f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb"
Sep 17 17:12:42.953136 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.953102    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb"} err="failed to get container status \"f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb\": rpc error: code = NotFound desc = could not find container \"f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb\": container with ID starting with f1186234682514adc9a0fcdf6d8f8042d6cd19a7b8ed25b6643f0e3b4439bffb not found: ID does not exist"
Sep 17 17:12:42.953136 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.953118    2566 scope.go:117] "RemoveContainer" containerID="15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515"
Sep 17 17:12:42.953391 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:12:42.953374    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515\": container with ID starting with 15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515 not found: ID does not exist" containerID="15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515"
Sep 17 17:12:42.953448 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.953397    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515"} err="failed to get container status \"15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515\": rpc error: code = NotFound desc = could not find container \"15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515\": container with ID starting with 15ff4025e6a52322b9b1c83b20f9298dcc2da4575087d915ac1108a1134fb515 not found: ID does not exist"
Sep 17 17:12:42.953448 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.953412    2566 scope.go:117] "RemoveContainer" containerID="2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0"
Sep 17 17:12:42.953635 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:12:42.953617    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0\": container with ID starting with 2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0 not found: ID does not exist" containerID="2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0"
Sep 17 17:12:42.953688 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:42.953642    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0"} err="failed to get container status \"2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0\": rpc error: code = NotFound desc = could not find container \"2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0\": container with ID starting with 2b0eb1f5fcdb4090eadb85e64aa571b05d274cfce5a8043c474e35c4affbbfd0 not found: ID does not exist"
Sep 17 17:12:43.594534 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.594506    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:12:43.640734 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.640706    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"]
Sep 17 17:12:43.641109 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641093    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container"
Sep 17 17:12:43.641198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641112    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container"
Sep 17 17:12:43.641198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641131    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kube-rbac-proxy"
Sep 17 17:12:43.641198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641141    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kube-rbac-proxy"
Sep 17 17:12:43.641198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641153    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kube-rbac-proxy"
Sep 17 17:12:43.641198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641162    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kube-rbac-proxy"
Sep 17 17:12:43.641198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641191    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="storage-initializer"
Sep 17 17:12:43.641198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641200    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="storage-initializer"
Sep 17 17:12:43.641585 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641211    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container"
Sep 17 17:12:43.641585 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641219    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container"
Sep 17 17:12:43.641585 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641231    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="storage-initializer"
Sep 17 17:12:43.641585 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641240    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="storage-initializer"
Sep 17 17:12:43.641585 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641339    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kserve-container"
Sep 17 17:12:43.641585 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641354    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa20d16-ee5d-40c9-88de-cc8969f2da58" containerName="kube-rbac-proxy"
Sep 17 17:12:43.641585 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641366    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kube-rbac-proxy"
Sep 17 17:12:43.641585 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.641379    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" containerName="kserve-container"
Sep 17 17:12:43.646656 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.646636    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.648847 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.648824    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 17:12:43.648959 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.648869    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:12:43.649605 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.649436    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:12:43.649605 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.649483    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 17:12:43.658452 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.649936    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:12:43.658452 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.649940    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:12:43.658452 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.649995    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:12:43.661791 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.661768    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"]
Sep 17 17:12:43.665931 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.665911    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"]
Sep 17 17:12:43.666046 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.666032    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.668618 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.668597    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 17:12:43.678204 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.678180    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"]
Sep 17 17:12:43.679605 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.679580    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:12:43.687569 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.687546    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.688824 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.687577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.688824 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.687615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ccaff9-796a-41ac-967b-a83cca96964b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.688824 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.687684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.688824 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.687737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.688824 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.687780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xd9\" (UniqueName: \"kubernetes.io/projected/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-api-access-s5xd9\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.688824 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.687823    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ccaff9-796a-41ac-967b-a83cca96964b-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.688824 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.687846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.688824 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.687869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.688824 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.687934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvh2\" (UniqueName: \"kubernetes.io/projected/73ccaff9-796a-41ac-967b-a83cca96964b-kube-api-access-dlvh2\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.789113 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ccaff9-796a-41ac-967b-a83cca96964b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.789309 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.789309 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.789309 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xd9\" (UniqueName: \"kubernetes.io/projected/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-api-access-s5xd9\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.789494 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ccaff9-796a-41ac-967b-a83cca96964b-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.789494 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.789494 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.789494 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvh2\" (UniqueName: \"kubernetes.io/projected/73ccaff9-796a-41ac-967b-a83cca96964b-kube-api-access-dlvh2\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.789494 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.789785 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.789785 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789567    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ccaff9-796a-41ac-967b-a83cca96964b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.789916 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.789995 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.789972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.790054 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.790016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.790112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.790071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.790112 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:12:43.790082    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 17:12:43.790217 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:12:43.790152    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-proxy-tls podName:aa9e7f1a-af20-4f8c-9e53-5d485529dc84 nodeName:}" failed. No retries permitted until 2026-09-17 17:12:44.290136402 +0000 UTC m=+1873.002143648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-proxy-tls") pod "isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" (UID: "aa9e7f1a-af20-4f8c-9e53-5d485529dc84") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 17:12:43.790217 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.790197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.791938 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.791917    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ccaff9-796a-41ac-967b-a83cca96964b-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.796780 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.796757    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xd9\" (UniqueName: \"kubernetes.io/projected/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-api-access-s5xd9\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:43.797469 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.797450    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvh2\" (UniqueName: \"kubernetes.io/projected/73ccaff9-796a-41ac-967b-a83cca96964b-kube-api-access-dlvh2\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-4ncz8\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:43.810401 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.810379    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2" path="/var/lib/kubelet/pods/9cd6115a-30d8-4e68-9c45-1c1ce4b4dcb2/volumes"
Sep 17 17:12:43.831005 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.830965    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:12:43.973536 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.973451    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:12:43.978350 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:43.978325    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:44.250950 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:44.250927    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"]
Sep 17 17:12:44.253609 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:12:44.253577    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ccaff9_796a_41ac_967b_a83cca96964b.slice/crio-42cb1edb1c0c155a71579d1864b498c676a7f09de3da396b3b0b8002cece2552 WatchSource:0}: Error finding container 42cb1edb1c0c155a71579d1864b498c676a7f09de3da396b3b0b8002cece2552: Status 404 returned error can't find the container with id 42cb1edb1c0c155a71579d1864b498c676a7f09de3da396b3b0b8002cece2552
Sep 17 17:12:44.293173 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:44.293147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:44.295414 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:44.295395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:44.561784 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:44.561706    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:44.836033 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:44.835972    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"]
Sep 17 17:12:44.838360 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:12:44.838328    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9e7f1a_af20_4f8c_9e53_5d485529dc84.slice/crio-1ef6b3d3503c84fa254b1ca5352ec62ea1e8f1c6d1378901fced3a610bbaede5 WatchSource:0}: Error finding container 1ef6b3d3503c84fa254b1ca5352ec62ea1e8f1c6d1378901fced3a610bbaede5: Status 404 returned error can't find the container with id 1ef6b3d3503c84fa254b1ca5352ec62ea1e8f1c6d1378901fced3a610bbaede5
Sep 17 17:12:44.938441 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:44.938406    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" event={"ID":"aa9e7f1a-af20-4f8c-9e53-5d485529dc84","Type":"ContainerStarted","Data":"ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7"}
Sep 17 17:12:44.938604 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:44.938472    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" event={"ID":"aa9e7f1a-af20-4f8c-9e53-5d485529dc84","Type":"ContainerStarted","Data":"1ef6b3d3503c84fa254b1ca5352ec62ea1e8f1c6d1378901fced3a610bbaede5"}
Sep 17 17:12:44.940367 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:44.940341    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" event={"ID":"73ccaff9-796a-41ac-967b-a83cca96964b","Type":"ContainerStarted","Data":"ffaa8be10b9418d386a1f85321270ace76959c64a76b9986d711641c75af11ca"}
Sep 17 17:12:44.940477 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:44.940386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" event={"ID":"73ccaff9-796a-41ac-967b-a83cca96964b","Type":"ContainerStarted","Data":"42cb1edb1c0c155a71579d1864b498c676a7f09de3da396b3b0b8002cece2552"}
Sep 17 17:12:48.840772 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:48.840733    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:12:48.960767 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:48.960735    2566 generic.go:356] "Generic (PLEG): container finished" podID="73ccaff9-796a-41ac-967b-a83cca96964b" containerID="ffaa8be10b9418d386a1f85321270ace76959c64a76b9986d711641c75af11ca" exitCode=0
Sep 17 17:12:48.960921 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:48.960803    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" event={"ID":"73ccaff9-796a-41ac-967b-a83cca96964b","Type":"ContainerDied","Data":"ffaa8be10b9418d386a1f85321270ace76959c64a76b9986d711641c75af11ca"}
Sep 17 17:12:48.962479 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:48.962455    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerID="ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7" exitCode=0
Sep 17 17:12:48.962565 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:48.962503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" event={"ID":"aa9e7f1a-af20-4f8c-9e53-5d485529dc84","Type":"ContainerDied","Data":"ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7"}
Sep 17 17:12:49.969312 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:49.969271    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" event={"ID":"aa9e7f1a-af20-4f8c-9e53-5d485529dc84","Type":"ContainerStarted","Data":"f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a"}
Sep 17 17:12:49.969772 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:49.969345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" event={"ID":"aa9e7f1a-af20-4f8c-9e53-5d485529dc84","Type":"ContainerStarted","Data":"b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb"}
Sep 17 17:12:49.969772 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:49.969653    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:49.969772 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:49.969710    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:49.971046 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:49.971004    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 17:12:49.971566 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:49.971545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" event={"ID":"73ccaff9-796a-41ac-967b-a83cca96964b","Type":"ContainerStarted","Data":"58ff9003f1d2af2a2b8f4dd889f6839a1fc15939c5f69dbf0dc78632642b3e17"}
Sep 17 17:12:49.971684 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:49.971586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" event={"ID":"73ccaff9-796a-41ac-967b-a83cca96964b","Type":"ContainerStarted","Data":"012c9cba3596a42b911d5b15949b8a44cf9b8d226a64bb02c8028567f29ccee3"}
Sep 17 17:12:49.971776 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:49.971758    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:49.971831 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:49.971802    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:49.972930 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:49.972910    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 17:12:49.988079 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:49.988040    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podStartSLOduration=3.384269843 podStartE2EDuration="6.988027209s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.603757366s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:12:49.986667648 +0000 UTC m=+1878.698674929" watchObservedRunningTime="2026-09-17 17:12:49.988027209 +0000 UTC m=+1878.700034477"
Sep 17 17:12:50.004341 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:50.004301    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podStartSLOduration=2.975220043 podStartE2EDuration="7.004289242s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.029069199s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:12:50.003485842 +0000 UTC m=+1878.715493109" watchObservedRunningTime="2026-09-17 17:12:50.004289242 +0000 UTC m=+1878.716296509"
Sep 17 17:12:50.976421 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:50.976381    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 17:12:50.976790 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:50.976502    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 17:12:53.973327 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:53.973230    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:12:55.049342 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:55.049299    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:12:55.981094 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:55.981059    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:12:55.981567 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:55.981544    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:12:55.981703 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:55.981586    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 17:12:55.981931 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:12:55.981909    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 17:13:05.982124 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:05.982082    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 17:13:05.982124 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:05.982093    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 17:13:15.982462 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:15.982425    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 17:13:15.982925 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:15.982426    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 17:13:25.981858 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:25.981820    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 17:13:25.982325 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:25.981964    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 17:13:34.048309 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:34.048247    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:13:35.982244 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:35.982205    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 17:13:35.982244 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:35.982205    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 17:13:39.169121 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:39.169083    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:13:39.780690 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:39.780653    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:13:45.982438 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:45.982401    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 17:13:45.982808 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:45.982413    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 17:13:55.981601 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:55.981566    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 17:13:55.982150 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:13:55.982034    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 17:14:05.982846 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:05.982815    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:14:05.983325 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:05.982913    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:14:13.652887 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:13.652848    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"]
Sep 17 17:14:13.653343 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:13.653133    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container" containerID="cri-o://b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb" gracePeriod=30
Sep 17 17:14:13.653343 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:13.653160    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kube-rbac-proxy" containerID="cri-o://f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a" gracePeriod=30
Sep 17 17:14:13.676560 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:13.676534    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:14:14.379656 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:14.379625    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerID="f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a" exitCode=2
Sep 17 17:14:14.379823 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:14.379676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" event={"ID":"aa9e7f1a-af20-4f8c-9e53-5d485529dc84","Type":"ContainerDied","Data":"f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a"}
Sep 17 17:14:15.977545 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:15.977499    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 17 17:14:15.981979 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:15.981944    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 17:14:17.701127 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.701101    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:14:17.749084 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.749060    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5xd9\" (UniqueName: \"kubernetes.io/projected/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-api-access-s5xd9\") pod \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") "
Sep 17 17:14:17.749235 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.749106    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kserve-provision-location\") pod \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") "
Sep 17 17:14:17.749235 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.749145    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-proxy-tls\") pod \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") "
Sep 17 17:14:17.749235 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.749187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-rbac-proxy-sar-config\") pod \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") "
Sep 17 17:14:17.749422 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.749241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-cabundle-cert\") pod \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\" (UID: \"aa9e7f1a-af20-4f8c-9e53-5d485529dc84\") "
Sep 17 17:14:17.749541 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.749453    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa9e7f1a-af20-4f8c-9e53-5d485529dc84" (UID: "aa9e7f1a-af20-4f8c-9e53-5d485529dc84"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:17.749655 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.749601    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa9e7f1a-af20-4f8c-9e53-5d485529dc84" (UID: "aa9e7f1a-af20-4f8c-9e53-5d485529dc84"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:17.749714 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.749667    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aa9e7f1a-af20-4f8c-9e53-5d485529dc84" (UID: "aa9e7f1a-af20-4f8c-9e53-5d485529dc84"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:17.751228 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.751210    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-api-access-s5xd9" (OuterVolumeSpecName: "kube-api-access-s5xd9") pod "aa9e7f1a-af20-4f8c-9e53-5d485529dc84" (UID: "aa9e7f1a-af20-4f8c-9e53-5d485529dc84"). InnerVolumeSpecName "kube-api-access-s5xd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:14:17.751340 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.751248    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa9e7f1a-af20-4f8c-9e53-5d485529dc84" (UID: "aa9e7f1a-af20-4f8c-9e53-5d485529dc84"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:14:17.850502 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.850480    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5xd9\" (UniqueName: \"kubernetes.io/projected/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-api-access-s5xd9\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:14:17.850502 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.850499    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:14:17.850633 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.850508    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:14:17.850633 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.850517    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:14:17.850633 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:17.850525    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa9e7f1a-af20-4f8c-9e53-5d485529dc84-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:14:18.399963 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.399923    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerID="b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb" exitCode=0
Sep 17 17:14:18.400162 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.399975    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"
Sep 17 17:14:18.400162 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.400006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" event={"ID":"aa9e7f1a-af20-4f8c-9e53-5d485529dc84","Type":"ContainerDied","Data":"b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb"}
Sep 17 17:14:18.400162 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.400082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq" event={"ID":"aa9e7f1a-af20-4f8c-9e53-5d485529dc84","Type":"ContainerDied","Data":"1ef6b3d3503c84fa254b1ca5352ec62ea1e8f1c6d1378901fced3a610bbaede5"}
Sep 17 17:14:18.400162 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.400120    2566 scope.go:117] "RemoveContainer" containerID="f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a"
Sep 17 17:14:18.409330 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.409319    2566 scope.go:117] "RemoveContainer" containerID="b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb"
Sep 17 17:14:18.417194 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.417179    2566 scope.go:117] "RemoveContainer" containerID="ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7"
Sep 17 17:14:18.418219 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.418198    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"]
Sep 17 17:14:18.422340 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.422241    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-nr7sq"]
Sep 17 17:14:18.425342 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.425326    2566 scope.go:117] "RemoveContainer" containerID="f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a"
Sep 17 17:14:18.425603 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:14:18.425585    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a\": container with ID starting with f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a not found: ID does not exist" containerID="f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a"
Sep 17 17:14:18.425656 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.425611    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a"} err="failed to get container status \"f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a\": rpc error: code = NotFound desc = could not find container \"f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a\": container with ID starting with f4c09b39184587148b669d90e1d86a54fe6355e08ade700ffc9897a9dd54ef2a not found: ID does not exist"
Sep 17 17:14:18.425656 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.425626    2566 scope.go:117] "RemoveContainer" containerID="b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb"
Sep 17 17:14:18.425878 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:14:18.425858    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb\": container with ID starting with b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb not found: ID does not exist" containerID="b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb"
Sep 17 17:14:18.425933 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.425884    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb"} err="failed to get container status \"b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb\": rpc error: code = NotFound desc = could not find container \"b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb\": container with ID starting with b27f266d3b582e4c9c4e96a0ba2a72f680d49c9f1d633b2c82cf5ba20b7b31cb not found: ID does not exist"
Sep 17 17:14:18.425933 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.425910    2566 scope.go:117] "RemoveContainer" containerID="ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7"
Sep 17 17:14:18.426139 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:14:18.426112    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7\": container with ID starting with ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7 not found: ID does not exist" containerID="ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7"
Sep 17 17:14:18.426214 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:18.426137    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7"} err="failed to get container status \"ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7\": rpc error: code = NotFound desc = could not find container \"ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7\": container with ID starting with ed3f31e9a8349a212b1d8030608de49acaf40a5027268e620fa362c2defd8cd7 not found: ID does not exist"
Sep 17 17:14:19.138794 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:19.138690    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:14:19.809586 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:19.809553    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" path="/var/lib/kubelet/pods/aa9e7f1a-af20-4f8c-9e53-5d485529dc84/volumes"
Sep 17 17:14:20.445065 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:20.445032    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:14:24.276573 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.276485    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:14:24.313307 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.313275    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"]
Sep 17 17:14:24.313628 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.313615    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container"
Sep 17 17:14:24.313628 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.313629    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container"
Sep 17 17:14:24.313721 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.313647    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="storage-initializer"
Sep 17 17:14:24.313721 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.313654    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="storage-initializer"
Sep 17 17:14:24.313721 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.313663    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kube-rbac-proxy"
Sep 17 17:14:24.313721 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.313669    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kube-rbac-proxy"
Sep 17 17:14:24.313870 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.313727    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kserve-container"
Sep 17 17:14:24.313870 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.313736    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9e7f1a-af20-4f8c-9e53-5d485529dc84" containerName="kube-rbac-proxy"
Sep 17 17:14:24.316859 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.316839    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.319132 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.319112    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 17:14:24.319233 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.319185    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 17:14:24.326170 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.326146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"]
Sep 17 17:14:24.373999 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.373971    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:14:24.399094 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.399069    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.399197 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.399105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/792e7430-53be-4202-86f5-86fc67611095-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.399197 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.399152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqwp\" (UniqueName: \"kubernetes.io/projected/792e7430-53be-4202-86f5-86fc67611095-kube-api-access-2xqwp\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.399197 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.399189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.399331 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.399280    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/792e7430-53be-4202-86f5-86fc67611095-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.500216 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.500182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.500360 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.500231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/792e7430-53be-4202-86f5-86fc67611095-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.500437 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.500409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqwp\" (UniqueName: \"kubernetes.io/projected/792e7430-53be-4202-86f5-86fc67611095-kube-api-access-2xqwp\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.500497 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.500450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.500554 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.500506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/792e7430-53be-4202-86f5-86fc67611095-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.500821 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.500802    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/792e7430-53be-4202-86f5-86fc67611095-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.500899 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.500833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.501129 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.501110    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.502979 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.502955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/792e7430-53be-4202-86f5-86fc67611095-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.508369 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.508344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqwp\" (UniqueName: \"kubernetes.io/projected/792e7430-53be-4202-86f5-86fc67611095-kube-api-access-2xqwp\") pod \"isvc-triton-predictor-5d7db58676-fcb62\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.629163 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.629114    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:14:24.900788 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:24.900763    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"]
Sep 17 17:14:24.902198 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:14:24.902176    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792e7430_53be_4202_86f5_86fc67611095.slice/crio-0201e7867da5ca011bf23a064a83833856179f3dcfb0bb0de1bf50dd56a8e32f WatchSource:0}: Error finding container 0201e7867da5ca011bf23a064a83833856179f3dcfb0bb0de1bf50dd56a8e32f: Status 404 returned error can't find the container with id 0201e7867da5ca011bf23a064a83833856179f3dcfb0bb0de1bf50dd56a8e32f
Sep 17 17:14:25.438294 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:25.438262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" event={"ID":"792e7430-53be-4202-86f5-86fc67611095","Type":"ContainerStarted","Data":"fd362fdff9b69a8f20e64c0da6acfdf11ba425ee03072f91f69b64cbaea38f04"}
Sep 17 17:14:25.438773 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:25.438332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" event={"ID":"792e7430-53be-4202-86f5-86fc67611095","Type":"ContainerStarted","Data":"0201e7867da5ca011bf23a064a83833856179f3dcfb0bb0de1bf50dd56a8e32f"}
Sep 17 17:14:29.458507 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:29.458424    2566 generic.go:356] "Generic (PLEG): container finished" podID="792e7430-53be-4202-86f5-86fc67611095" containerID="fd362fdff9b69a8f20e64c0da6acfdf11ba425ee03072f91f69b64cbaea38f04" exitCode=0
Sep 17 17:14:29.458992 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:29.458504    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" event={"ID":"792e7430-53be-4202-86f5-86fc67611095","Type":"ContainerDied","Data":"fd362fdff9b69a8f20e64c0da6acfdf11ba425ee03072f91f69b64cbaea38f04"}
Sep 17 17:14:29.459846 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:14:29.459826    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:15:02.555478 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:02.555385    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:15:33.749234 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:33.749198    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"]
Sep 17 17:15:33.749843 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:33.749506    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container" containerID="cri-o://012c9cba3596a42b911d5b15949b8a44cf9b8d226a64bb02c8028567f29ccee3" gracePeriod=30
Sep 17 17:15:33.749843 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:33.749530    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kube-rbac-proxy" containerID="cri-o://58ff9003f1d2af2a2b8f4dd889f6839a1fc15939c5f69dbf0dc78632642b3e17" gracePeriod=30
Sep 17 17:15:33.842765 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:33.842729    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:15:35.977651 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:35.977609    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Sep 17 17:15:35.981914 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:35.981885    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 17:15:38.925064 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:38.924908    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:15:38.997673 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:38.997503    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:15:39.050239 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.050205    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"]
Sep 17 17:15:39.060383 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.060361    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.064163 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.063137    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 17:15:39.064163 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.063393    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 17:15:39.064918 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.064891    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"]
Sep 17 17:15:39.094697 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.094672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.094829 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.094718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.094829 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.094774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c94378-12ba-4972-aec7-4c7bf03a4dac-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.094976 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.094889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c94378-12ba-4972-aec7-4c7bf03a4dac-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.094976 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.094911    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdhxp\" (UniqueName: \"kubernetes.io/projected/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-api-access-hdhxp\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.157441 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.157408    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:15:39.196016 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.195939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c94378-12ba-4972-aec7-4c7bf03a4dac-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.196016 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.195980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdhxp\" (UniqueName: \"kubernetes.io/projected/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-api-access-hdhxp\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.196218 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.196036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.196218 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.196065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.196218 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.196121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c94378-12ba-4972-aec7-4c7bf03a4dac-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.196417 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:15:39.196287    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 17:15:39.196417 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:15:39.196363    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86c94378-12ba-4972-aec7-4c7bf03a4dac-proxy-tls podName:86c94378-12ba-4972-aec7-4c7bf03a4dac nodeName:}" failed. No retries permitted until 2026-09-17 17:15:39.696341471 +0000 UTC m=+2048.408348735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/86c94378-12ba-4972-aec7-4c7bf03a4dac-proxy-tls") pod "isvc-canary-stop-predictor-67d6bcd4b5-47hdv" (UID: "86c94378-12ba-4972-aec7-4c7bf03a4dac") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 17:15:39.196535 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.196455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c94378-12ba-4972-aec7-4c7bf03a4dac-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.197009 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.196987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.197113 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.197094    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.204876 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.204853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdhxp\" (UniqueName: \"kubernetes.io/projected/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-api-access-hdhxp\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.354630 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.354595    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:15:39.701152 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.701120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c94378-12ba-4972-aec7-4c7bf03a4dac-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.703896 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.703868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c94378-12ba-4972-aec7-4c7bf03a4dac-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-47hdv\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:39.974152 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:39.974071    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:15:40.977052 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:40.977011    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Sep 17 17:15:42.659274 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:42.659233    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"]
Sep 17 17:15:42.917461 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:42.917371    2566 generic.go:356] "Generic (PLEG): container finished" podID="73ccaff9-796a-41ac-967b-a83cca96964b" containerID="58ff9003f1d2af2a2b8f4dd889f6839a1fc15939c5f69dbf0dc78632642b3e17" exitCode=2
Sep 17 17:15:42.917461 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:42.917401    2566 generic.go:356] "Generic (PLEG): container finished" podID="73ccaff9-796a-41ac-967b-a83cca96964b" containerID="012c9cba3596a42b911d5b15949b8a44cf9b8d226a64bb02c8028567f29ccee3" exitCode=0
Sep 17 17:15:42.917461 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:42.917438    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" event={"ID":"73ccaff9-796a-41ac-967b-a83cca96964b","Type":"ContainerDied","Data":"58ff9003f1d2af2a2b8f4dd889f6839a1fc15939c5f69dbf0dc78632642b3e17"}
Sep 17 17:15:42.917768 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:42.917527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" event={"ID":"73ccaff9-796a-41ac-967b-a83cca96964b","Type":"ContainerDied","Data":"012c9cba3596a42b911d5b15949b8a44cf9b8d226a64bb02c8028567f29ccee3"}
Sep 17 17:15:43.298417 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.298384    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:15:43.330614 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.330578    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-kube-rbac-proxy-sar-config\") pod \"73ccaff9-796a-41ac-967b-a83cca96964b\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") "
Sep 17 17:15:43.330809 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.330659    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ccaff9-796a-41ac-967b-a83cca96964b-kserve-provision-location\") pod \"73ccaff9-796a-41ac-967b-a83cca96964b\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") "
Sep 17 17:15:43.330809 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.330790    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-cabundle-cert\") pod \"73ccaff9-796a-41ac-967b-a83cca96964b\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") "
Sep 17 17:15:43.330959 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.330929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlvh2\" (UniqueName: \"kubernetes.io/projected/73ccaff9-796a-41ac-967b-a83cca96964b-kube-api-access-dlvh2\") pod \"73ccaff9-796a-41ac-967b-a83cca96964b\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") "
Sep 17 17:15:43.331019 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.330968    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ccaff9-796a-41ac-967b-a83cca96964b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73ccaff9-796a-41ac-967b-a83cca96964b" (UID: "73ccaff9-796a-41ac-967b-a83cca96964b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:15:43.331019 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.330972    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ccaff9-796a-41ac-967b-a83cca96964b-proxy-tls\") pod \"73ccaff9-796a-41ac-967b-a83cca96964b\" (UID: \"73ccaff9-796a-41ac-967b-a83cca96964b\") "
Sep 17 17:15:43.331142 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.331036    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73ccaff9-796a-41ac-967b-a83cca96964b" (UID: "73ccaff9-796a-41ac-967b-a83cca96964b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:15:43.331214 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.331183    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73ccaff9-796a-41ac-967b-a83cca96964b" (UID: "73ccaff9-796a-41ac-967b-a83cca96964b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:15:43.331331 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.331310    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:15:43.331409 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.331335    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ccaff9-796a-41ac-967b-a83cca96964b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:15:43.331409 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.331350    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ccaff9-796a-41ac-967b-a83cca96964b-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:15:43.334370 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.334313    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ccaff9-796a-41ac-967b-a83cca96964b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73ccaff9-796a-41ac-967b-a83cca96964b" (UID: "73ccaff9-796a-41ac-967b-a83cca96964b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:15:43.334543 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.334375    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ccaff9-796a-41ac-967b-a83cca96964b-kube-api-access-dlvh2" (OuterVolumeSpecName: "kube-api-access-dlvh2") pod "73ccaff9-796a-41ac-967b-a83cca96964b" (UID: "73ccaff9-796a-41ac-967b-a83cca96964b"). InnerVolumeSpecName "kube-api-access-dlvh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:15:43.432534 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.432495    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlvh2\" (UniqueName: \"kubernetes.io/projected/73ccaff9-796a-41ac-967b-a83cca96964b-kube-api-access-dlvh2\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:15:43.432534 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.432528    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ccaff9-796a-41ac-967b-a83cca96964b-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:15:43.926567 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.926529    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8" event={"ID":"73ccaff9-796a-41ac-967b-a83cca96964b","Type":"ContainerDied","Data":"42cb1edb1c0c155a71579d1864b498c676a7f09de3da396b3b0b8002cece2552"}
Sep 17 17:15:43.927064 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.926585    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"
Sep 17 17:15:43.927064 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.926614    2566 scope.go:117] "RemoveContainer" containerID="58ff9003f1d2af2a2b8f4dd889f6839a1fc15939c5f69dbf0dc78632642b3e17"
Sep 17 17:15:43.946438 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.946407    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"]
Sep 17 17:15:43.949331 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:43.949286    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-4ncz8"]
Sep 17 17:15:44.386861 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:15:44.386823    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c94378_12ba_4972_aec7_4c7bf03a4dac.slice/crio-926e66e3602781890185a78343432c20d62c93e8b0866ddb24fd28f160a0fe5c WatchSource:0}: Error finding container 926e66e3602781890185a78343432c20d62c93e8b0866ddb24fd28f160a0fe5c: Status 404 returned error can't find the container with id 926e66e3602781890185a78343432c20d62c93e8b0866ddb24fd28f160a0fe5c
Sep 17 17:15:44.473942 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:44.473918    2566 scope.go:117] "RemoveContainer" containerID="012c9cba3596a42b911d5b15949b8a44cf9b8d226a64bb02c8028567f29ccee3"
Sep 17 17:15:44.936643 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:44.936602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" event={"ID":"86c94378-12ba-4972-aec7-4c7bf03a4dac","Type":"ContainerStarted","Data":"926e66e3602781890185a78343432c20d62c93e8b0866ddb24fd28f160a0fe5c"}
Sep 17 17:15:45.812125 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:45.812079    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" path="/var/lib/kubelet/pods/73ccaff9-796a-41ac-967b-a83cca96964b/volumes"
Sep 17 17:15:48.596355 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:48.596328    2566 scope.go:117] "RemoveContainer" containerID="ffaa8be10b9418d386a1f85321270ace76959c64a76b9986d711641c75af11ca"
Sep 17 17:15:53.992917 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:15:53.992877    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" event={"ID":"86c94378-12ba-4972-aec7-4c7bf03a4dac","Type":"ContainerStarted","Data":"4caa2ec79a47ca2dfbb8b7acfeedab5923a4bc6984ea8d5de8565c535b1faa1c"}
Sep 17 17:16:22.145885 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:22.145850    2566 generic.go:356] "Generic (PLEG): container finished" podID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerID="4caa2ec79a47ca2dfbb8b7acfeedab5923a4bc6984ea8d5de8565c535b1faa1c" exitCode=0
Sep 17 17:16:22.146367 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:22.145936    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" event={"ID":"86c94378-12ba-4972-aec7-4c7bf03a4dac","Type":"ContainerDied","Data":"4caa2ec79a47ca2dfbb8b7acfeedab5923a4bc6984ea8d5de8565c535b1faa1c"}
Sep 17 17:16:23.153762 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:23.153727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" event={"ID":"86c94378-12ba-4972-aec7-4c7bf03a4dac","Type":"ContainerStarted","Data":"f86edd44da169bf7390af94cca34c719d6865b8fb79a7f5c50f5e8adf4bfe1ef"}
Sep 17 17:16:23.154171 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:23.153818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" event={"ID":"86c94378-12ba-4972-aec7-4c7bf03a4dac","Type":"ContainerStarted","Data":"19f9577a5315bd3d2f2ac46c5ceb1f0971b92a13cc09a2dd55268cbfb7e7bead"}
Sep 17 17:16:23.154171 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:23.154114    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:16:23.172659 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:23.172607    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podStartSLOduration=39.695692392 podStartE2EDuration="44.172592385s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.476899993s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:16:23.171152161 +0000 UTC m=+2091.883159428" watchObservedRunningTime="2026-09-17 17:16:23.172592385 +0000 UTC m=+2091.884599691"
Sep 17 17:16:24.160830 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:24.160794    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" event={"ID":"792e7430-53be-4202-86f5-86fc67611095","Type":"ContainerStarted","Data":"32b432058ef58d852919c1f8b40b58bc6ba25e97bd6b1e589af82b06397489d5"}
Sep 17 17:16:24.161198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:24.160862    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" event={"ID":"792e7430-53be-4202-86f5-86fc67611095","Type":"ContainerStarted","Data":"0d192ca472fec8307507afdd6157813ec9e5445576cafe47cf737b3449991750"}
Sep 17 17:16:24.161198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:24.161080    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:16:24.162518 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:24.162490    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 17:16:24.178934 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:24.178888    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" podStartSLOduration=1.954891304 podStartE2EDuration="2m0.178873028s" totalImagesPullingTime="1m54.097056113s" totalInitContainerRuntime="4.126925611s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:16:24.177779485 +0000 UTC m=+2092.889786753" watchObservedRunningTime="2026-09-17 17:16:24.178873028 +0000 UTC m=+2092.890880286"
Sep 17 17:16:25.167418 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:25.167373    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 17:16:25.167418 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:25.167407    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:16:25.167828 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:25.167462    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:16:25.168521 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:25.168497    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 17:16:26.171947 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:26.171899    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 17:16:30.171701 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:30.171666    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:16:30.172311 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:30.172284    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 17:16:31.176335 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:31.176310    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:16:31.177034 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:31.177013    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:16:35.881684 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:35.881653    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"]
Sep 17 17:16:35.882165 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:35.881887    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="kserve-container" containerID="cri-o://0d192ca472fec8307507afdd6157813ec9e5445576cafe47cf737b3449991750" gracePeriod=30
Sep 17 17:16:35.882165 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:35.881940    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="kube-rbac-proxy" containerID="cri-o://32b432058ef58d852919c1f8b40b58bc6ba25e97bd6b1e589af82b06397489d5" gracePeriod=30
Sep 17 17:16:35.903334 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:35.903310    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:16:36.172120 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:36.172049    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Sep 17 17:16:36.222046 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:36.222020    2566 generic.go:356] "Generic (PLEG): container finished" podID="792e7430-53be-4202-86f5-86fc67611095" containerID="32b432058ef58d852919c1f8b40b58bc6ba25e97bd6b1e589af82b06397489d5" exitCode=2
Sep 17 17:16:36.222175 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:36.222061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" event={"ID":"792e7430-53be-4202-86f5-86fc67611095","Type":"ContainerDied","Data":"32b432058ef58d852919c1f8b40b58bc6ba25e97bd6b1e589af82b06397489d5"}
Sep 17 17:16:38.235823 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.235780    2566 generic.go:356] "Generic (PLEG): container finished" podID="792e7430-53be-4202-86f5-86fc67611095" containerID="0d192ca472fec8307507afdd6157813ec9e5445576cafe47cf737b3449991750" exitCode=0
Sep 17 17:16:38.236164 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.235837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" event={"ID":"792e7430-53be-4202-86f5-86fc67611095","Type":"ContainerDied","Data":"0d192ca472fec8307507afdd6157813ec9e5445576cafe47cf737b3449991750"}
Sep 17 17:16:38.347239 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.347218    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:16:38.419467 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.419400    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xqwp\" (UniqueName: \"kubernetes.io/projected/792e7430-53be-4202-86f5-86fc67611095-kube-api-access-2xqwp\") pod \"792e7430-53be-4202-86f5-86fc67611095\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") "
Sep 17 17:16:38.419467 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.419443    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/792e7430-53be-4202-86f5-86fc67611095-kserve-provision-location\") pod \"792e7430-53be-4202-86f5-86fc67611095\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") "
Sep 17 17:16:38.419664 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.419483    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-cabundle-cert\") pod \"792e7430-53be-4202-86f5-86fc67611095\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") "
Sep 17 17:16:38.419664 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.419549    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-kube-rbac-proxy-sar-config\") pod \"792e7430-53be-4202-86f5-86fc67611095\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") "
Sep 17 17:16:38.419664 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.419613    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/792e7430-53be-4202-86f5-86fc67611095-proxy-tls\") pod \"792e7430-53be-4202-86f5-86fc67611095\" (UID: \"792e7430-53be-4202-86f5-86fc67611095\") "
Sep 17 17:16:38.419877 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.419847    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "792e7430-53be-4202-86f5-86fc67611095" (UID: "792e7430-53be-4202-86f5-86fc67611095"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:16:38.419947 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.419902    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792e7430-53be-4202-86f5-86fc67611095-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "792e7430-53be-4202-86f5-86fc67611095" (UID: "792e7430-53be-4202-86f5-86fc67611095"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:16:38.419947 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.419914    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "792e7430-53be-4202-86f5-86fc67611095" (UID: "792e7430-53be-4202-86f5-86fc67611095"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:16:38.421511 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.421488    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792e7430-53be-4202-86f5-86fc67611095-kube-api-access-2xqwp" (OuterVolumeSpecName: "kube-api-access-2xqwp") pod "792e7430-53be-4202-86f5-86fc67611095" (UID: "792e7430-53be-4202-86f5-86fc67611095"). InnerVolumeSpecName "kube-api-access-2xqwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:16:38.421661 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.421641    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792e7430-53be-4202-86f5-86fc67611095-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "792e7430-53be-4202-86f5-86fc67611095" (UID: "792e7430-53be-4202-86f5-86fc67611095"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:16:38.520344 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.520323    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:16:38.520344 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.520342    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/792e7430-53be-4202-86f5-86fc67611095-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:16:38.520487 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.520353    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xqwp\" (UniqueName: \"kubernetes.io/projected/792e7430-53be-4202-86f5-86fc67611095-kube-api-access-2xqwp\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:16:38.520487 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.520362    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/792e7430-53be-4202-86f5-86fc67611095-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:16:38.520487 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:38.520372    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/792e7430-53be-4202-86f5-86fc67611095-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:16:39.242881 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:39.242848    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"
Sep 17 17:16:39.243364 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:39.242881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62" event={"ID":"792e7430-53be-4202-86f5-86fc67611095","Type":"ContainerDied","Data":"0201e7867da5ca011bf23a064a83833856179f3dcfb0bb0de1bf50dd56a8e32f"}
Sep 17 17:16:39.243364 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:39.242954    2566 scope.go:117] "RemoveContainer" containerID="32b432058ef58d852919c1f8b40b58bc6ba25e97bd6b1e589af82b06397489d5"
Sep 17 17:16:39.252595 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:39.252576    2566 scope.go:117] "RemoveContainer" containerID="0d192ca472fec8307507afdd6157813ec9e5445576cafe47cf737b3449991750"
Sep 17 17:16:39.260146 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:39.260128    2566 scope.go:117] "RemoveContainer" containerID="fd362fdff9b69a8f20e64c0da6acfdf11ba425ee03072f91f69b64cbaea38f04"
Sep 17 17:16:39.266917 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:39.266894    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"]
Sep 17 17:16:39.272699 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:39.272677    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-fcb62"]
Sep 17 17:16:39.809363 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:39.809331    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="792e7430-53be-4202-86f5-86fc67611095" path="/var/lib/kubelet/pods/792e7430-53be-4202-86f5-86fc67611095/volumes"
Sep 17 17:16:40.172942 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:40.172858    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 17:16:41.025231 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.025190    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:16:41.034926 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.034900    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"]
Sep 17 17:16:41.035274 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035262    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="storage-initializer"
Sep 17 17:16:41.035322 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035276    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="storage-initializer"
Sep 17 17:16:41.035322 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035286    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kube-rbac-proxy"
Sep 17 17:16:41.035322 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035291    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kube-rbac-proxy"
Sep 17 17:16:41.035322 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035308    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="kserve-container"
Sep 17 17:16:41.035322 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035314    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="kserve-container"
Sep 17 17:16:41.035484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035324    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container"
Sep 17 17:16:41.035484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035329    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container"
Sep 17 17:16:41.035484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035335    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="kube-rbac-proxy"
Sep 17 17:16:41.035484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035341    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="kube-rbac-proxy"
Sep 17 17:16:41.035484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035351    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="storage-initializer"
Sep 17 17:16:41.035484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035356    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="storage-initializer"
Sep 17 17:16:41.035484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035409    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kserve-container"
Sep 17 17:16:41.035484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035419    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="kserve-container"
Sep 17 17:16:41.035484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035425    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="792e7430-53be-4202-86f5-86fc67611095" containerName="kube-rbac-proxy"
Sep 17 17:16:41.035484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.035431    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73ccaff9-796a-41ac-967b-a83cca96964b" containerName="kube-rbac-proxy"
Sep 17 17:16:41.066242 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.066219    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"]
Sep 17 17:16:41.066378 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.066344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.068838 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.068804    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 17:16:41.068838 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.068829    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 17:16:41.139001 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.138968    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.139152 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.139005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmwl\" (UniqueName: \"kubernetes.io/projected/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-api-access-zlmwl\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.139152 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.139033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46383b3e-ad21-4d92-a14f-e265821d0d7f-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.139152 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.139067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.139152 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.139098    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46383b3e-ad21-4d92-a14f-e265821d0d7f-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.218765 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.218728    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:16:41.240139 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.240114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.240296 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.240162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmwl\" (UniqueName: \"kubernetes.io/projected/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-api-access-zlmwl\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.240296 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.240210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46383b3e-ad21-4d92-a14f-e265821d0d7f-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.240296 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.240237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.240296 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.240287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46383b3e-ad21-4d92-a14f-e265821d0d7f-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.240820 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.240794    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46383b3e-ad21-4d92-a14f-e265821d0d7f-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.240961 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.240943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.241022 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.240967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.242613 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.242595    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46383b3e-ad21-4d92-a14f-e265821d0d7f-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.248025 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.248002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmwl\" (UniqueName: \"kubernetes.io/projected/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-api-access-zlmwl\") pod \"isvc-xgboost-predictor-7566d65bf7-5sqxw\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.376476 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.376400    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:16:41.678349 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:41.678325    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"]
Sep 17 17:16:41.680367 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:16:41.680339    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46383b3e_ad21_4d92_a14f_e265821d0d7f.slice/crio-ae892886dc4b5d820d8f3dc9ab59d8a2b501a376aac8553bd78f78db35c44733 WatchSource:0}: Error finding container ae892886dc4b5d820d8f3dc9ab59d8a2b501a376aac8553bd78f78db35c44733: Status 404 returned error can't find the container with id ae892886dc4b5d820d8f3dc9ab59d8a2b501a376aac8553bd78f78db35c44733
Sep 17 17:16:42.261188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:42.261147    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" event={"ID":"46383b3e-ad21-4d92-a14f-e265821d0d7f","Type":"ContainerStarted","Data":"211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383"}
Sep 17 17:16:42.261632 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:42.261223    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" event={"ID":"46383b3e-ad21-4d92-a14f-e265821d0d7f","Type":"ContainerStarted","Data":"ae892886dc4b5d820d8f3dc9ab59d8a2b501a376aac8553bd78f78db35c44733"}
Sep 17 17:16:46.282248 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:46.282215    2566 generic.go:356] "Generic (PLEG): container finished" podID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerID="211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383" exitCode=0
Sep 17 17:16:46.282620 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:46.282289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" event={"ID":"46383b3e-ad21-4d92-a14f-e265821d0d7f","Type":"ContainerDied","Data":"211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383"}
Sep 17 17:16:50.173024 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:50.172977    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 17:16:58.022309 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:16:58.022243    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:17:00.172282 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:00.172223    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 17:17:01.926603 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:01.926565    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"]
Sep 17 17:17:01.927074 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:01.926863    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container" containerID="cri-o://19f9577a5315bd3d2f2ac46c5ceb1f0971b92a13cc09a2dd55268cbfb7e7bead" gracePeriod=30
Sep 17 17:17:01.927074 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:01.927029    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kube-rbac-proxy" containerID="cri-o://f86edd44da169bf7390af94cca34c719d6865b8fb79a7f5c50f5e8adf4bfe1ef" gracePeriod=30
Sep 17 17:17:01.927247 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:01.927213    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 17:17:05.167959 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:05.167916    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 17 17:17:08.404464 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:08.404384    2566 generic.go:356] "Generic (PLEG): container finished" podID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerID="f86edd44da169bf7390af94cca34c719d6865b8fb79a7f5c50f5e8adf4bfe1ef" exitCode=2
Sep 17 17:17:08.404839 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:08.404459    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" event={"ID":"86c94378-12ba-4972-aec7-4c7bf03a4dac","Type":"ContainerDied","Data":"f86edd44da169bf7390af94cca34c719d6865b8fb79a7f5c50f5e8adf4bfe1ef"}
Sep 17 17:17:08.406570 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:08.406545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" event={"ID":"46383b3e-ad21-4d92-a14f-e265821d0d7f","Type":"ContainerStarted","Data":"0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583"}
Sep 17 17:17:08.406684 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:08.406594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" event={"ID":"46383b3e-ad21-4d92-a14f-e265821d0d7f","Type":"ContainerStarted","Data":"76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b"}
Sep 17 17:17:08.406794 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:08.406779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:17:08.424717 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:08.424666    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" podStartSLOduration=1.6698980300000001 podStartE2EDuration="27.424652827s" totalImagesPullingTime="21.736338437s" totalInitContainerRuntime="4.01841636s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:17:08.423728698 +0000 UTC m=+2137.135735979" watchObservedRunningTime="2026-09-17 17:17:08.424652827 +0000 UTC m=+2137.136660096"
Sep 17 17:17:08.968490 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:08.968457    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:17:09.046358 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:09.046323    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:17:09.411888 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:09.411858    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:17:09.413104 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:09.413076    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 17:17:10.167854 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:10.167815    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 17 17:17:10.416766 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:10.416733    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 17:17:11.927898 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:11.927854    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 17:17:15.167456 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:15.167418    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 17 17:17:15.167824 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:15.167531    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:17:15.424298 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:15.424194    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:17:15.424799 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:15.424774    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 17:17:19.197193 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:19.197153    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:17:19.296979 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:19.296934    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:17:20.168189 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:20.168141    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 17 17:17:21.927030 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:21.926988    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 17:17:25.167923 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:25.167835    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 17 17:17:25.425108 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:25.425008    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 17:17:30.168436 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:30.168394    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 17 17:17:31.927412 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:31.927363    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 17:17:32.533330 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.533301    2566 generic.go:356] "Generic (PLEG): container finished" podID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerID="19f9577a5315bd3d2f2ac46c5ceb1f0971b92a13cc09a2dd55268cbfb7e7bead" exitCode=137
Sep 17 17:17:32.533465 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.533375    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" event={"ID":"86c94378-12ba-4972-aec7-4c7bf03a4dac","Type":"ContainerDied","Data":"19f9577a5315bd3d2f2ac46c5ceb1f0971b92a13cc09a2dd55268cbfb7e7bead"}
Sep 17 17:17:32.575651 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.575628    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:17:32.663774 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.663753    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-rbac-proxy-sar-config\") pod \"86c94378-12ba-4972-aec7-4c7bf03a4dac\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") "
Sep 17 17:17:32.663903 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.663785    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c94378-12ba-4972-aec7-4c7bf03a4dac-kserve-provision-location\") pod \"86c94378-12ba-4972-aec7-4c7bf03a4dac\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") "
Sep 17 17:17:32.663903 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.663815    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c94378-12ba-4972-aec7-4c7bf03a4dac-proxy-tls\") pod \"86c94378-12ba-4972-aec7-4c7bf03a4dac\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") "
Sep 17 17:17:32.663903 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.663867    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-cabundle-cert\") pod \"86c94378-12ba-4972-aec7-4c7bf03a4dac\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") "
Sep 17 17:17:32.663903 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.663896    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdhxp\" (UniqueName: \"kubernetes.io/projected/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-api-access-hdhxp\") pod \"86c94378-12ba-4972-aec7-4c7bf03a4dac\" (UID: \"86c94378-12ba-4972-aec7-4c7bf03a4dac\") "
Sep 17 17:17:32.664173 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.664148    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c94378-12ba-4972-aec7-4c7bf03a4dac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86c94378-12ba-4972-aec7-4c7bf03a4dac" (UID: "86c94378-12ba-4972-aec7-4c7bf03a4dac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:17:32.664279 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.664187    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86c94378-12ba-4972-aec7-4c7bf03a4dac" (UID: "86c94378-12ba-4972-aec7-4c7bf03a4dac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:17:32.664279 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.664229    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86c94378-12ba-4972-aec7-4c7bf03a4dac" (UID: "86c94378-12ba-4972-aec7-4c7bf03a4dac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:17:32.665967 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.665947    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c94378-12ba-4972-aec7-4c7bf03a4dac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86c94378-12ba-4972-aec7-4c7bf03a4dac" (UID: "86c94378-12ba-4972-aec7-4c7bf03a4dac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:17:32.666038 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.666012    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-api-access-hdhxp" (OuterVolumeSpecName: "kube-api-access-hdhxp") pod "86c94378-12ba-4972-aec7-4c7bf03a4dac" (UID: "86c94378-12ba-4972-aec7-4c7bf03a4dac"). InnerVolumeSpecName "kube-api-access-hdhxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:17:32.764663 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.764602    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdhxp\" (UniqueName: \"kubernetes.io/projected/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-api-access-hdhxp\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:17:32.764663 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.764635    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:17:32.764663 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.764644    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c94378-12ba-4972-aec7-4c7bf03a4dac-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:17:32.764663 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.764653    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c94378-12ba-4972-aec7-4c7bf03a4dac-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:17:32.764663 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:32.764661    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c94378-12ba-4972-aec7-4c7bf03a4dac-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:17:33.298931 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:33.298898    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:17:33.541432 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:33.541404    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"
Sep 17 17:17:33.541591 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:33.541433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv" event={"ID":"86c94378-12ba-4972-aec7-4c7bf03a4dac","Type":"ContainerDied","Data":"926e66e3602781890185a78343432c20d62c93e8b0866ddb24fd28f160a0fe5c"}
Sep 17 17:17:33.541591 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:33.541512    2566 scope.go:117] "RemoveContainer" containerID="f86edd44da169bf7390af94cca34c719d6865b8fb79a7f5c50f5e8adf4bfe1ef"
Sep 17 17:17:33.550689 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:33.550670    2566 scope.go:117] "RemoveContainer" containerID="19f9577a5315bd3d2f2ac46c5ceb1f0971b92a13cc09a2dd55268cbfb7e7bead"
Sep 17 17:17:33.557900 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:33.557883    2566 scope.go:117] "RemoveContainer" containerID="4caa2ec79a47ca2dfbb8b7acfeedab5923a4bc6984ea8d5de8565c535b1faa1c"
Sep 17 17:17:33.564467 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:33.564447    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"]
Sep 17 17:17:33.568463 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:33.568442    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-47hdv"]
Sep 17 17:17:33.808888 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:33.808827    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" path="/var/lib/kubelet/pods/86c94378-12ba-4972-aec7-4c7bf03a4dac/volumes"
Sep 17 17:17:35.425708 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:35.425672    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 17:17:45.425124 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:45.425085    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 17:17:55.425075 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:17:55.425032    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 17:18:03.083473 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:03.083434    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:18:05.413559 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:05.413521    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:18:05.425175 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:05.425145    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 17:18:15.425404 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:15.425367    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:18:21.556410 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:21.556374    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"]
Sep 17 17:18:21.556860 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:21.556689    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container" containerID="cri-o://76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b" gracePeriod=30
Sep 17 17:18:21.556860 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:21.556733    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kube-rbac-proxy" containerID="cri-o://0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583" gracePeriod=30
Sep 17 17:18:21.576578 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:21.576547    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:18:21.768739 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:21.768715    2566 generic.go:356] "Generic (PLEG): container finished" podID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerID="0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583" exitCode=2
Sep 17 17:18:21.768841 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:21.768788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" event={"ID":"46383b3e-ad21-4d92-a14f-e265821d0d7f","Type":"ContainerDied","Data":"0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583"}
Sep 17 17:18:25.200372 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.200350    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:18:25.382068 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.382003    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-cabundle-cert\") pod \"46383b3e-ad21-4d92-a14f-e265821d0d7f\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") "
Sep 17 17:18:25.382068 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.382043    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlmwl\" (UniqueName: \"kubernetes.io/projected/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-api-access-zlmwl\") pod \"46383b3e-ad21-4d92-a14f-e265821d0d7f\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") "
Sep 17 17:18:25.382316 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.382081    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-rbac-proxy-sar-config\") pod \"46383b3e-ad21-4d92-a14f-e265821d0d7f\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") "
Sep 17 17:18:25.382316 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.382100    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46383b3e-ad21-4d92-a14f-e265821d0d7f-kserve-provision-location\") pod \"46383b3e-ad21-4d92-a14f-e265821d0d7f\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") "
Sep 17 17:18:25.382316 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.382122    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46383b3e-ad21-4d92-a14f-e265821d0d7f-proxy-tls\") pod \"46383b3e-ad21-4d92-a14f-e265821d0d7f\" (UID: \"46383b3e-ad21-4d92-a14f-e265821d0d7f\") "
Sep 17 17:18:25.382493 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.382412    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "46383b3e-ad21-4d92-a14f-e265821d0d7f" (UID: "46383b3e-ad21-4d92-a14f-e265821d0d7f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:18:25.382493 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.382428    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46383b3e-ad21-4d92-a14f-e265821d0d7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "46383b3e-ad21-4d92-a14f-e265821d0d7f" (UID: "46383b3e-ad21-4d92-a14f-e265821d0d7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:18:25.382493 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.382475    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "46383b3e-ad21-4d92-a14f-e265821d0d7f" (UID: "46383b3e-ad21-4d92-a14f-e265821d0d7f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:18:25.384306 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.384285    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-api-access-zlmwl" (OuterVolumeSpecName: "kube-api-access-zlmwl") pod "46383b3e-ad21-4d92-a14f-e265821d0d7f" (UID: "46383b3e-ad21-4d92-a14f-e265821d0d7f"). InnerVolumeSpecName "kube-api-access-zlmwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:18:25.384306 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.384297    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46383b3e-ad21-4d92-a14f-e265821d0d7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46383b3e-ad21-4d92-a14f-e265821d0d7f" (UID: "46383b3e-ad21-4d92-a14f-e265821d0d7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:18:25.483019 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.482987    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:18:25.483019 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.483020    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46383b3e-ad21-4d92-a14f-e265821d0d7f-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:18:25.483147 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.483030    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46383b3e-ad21-4d92-a14f-e265821d0d7f-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:18:25.483147 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.483040    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46383b3e-ad21-4d92-a14f-e265821d0d7f-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:18:25.483147 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.483050    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlmwl\" (UniqueName: \"kubernetes.io/projected/46383b3e-ad21-4d92-a14f-e265821d0d7f-kube-api-access-zlmwl\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:18:25.789759 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.789727    2566 generic.go:356] "Generic (PLEG): container finished" podID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerID="76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b" exitCode=0
Sep 17 17:18:25.789898 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.789785    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"
Sep 17 17:18:25.789898 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.789815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" event={"ID":"46383b3e-ad21-4d92-a14f-e265821d0d7f","Type":"ContainerDied","Data":"76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b"}
Sep 17 17:18:25.789898 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.789873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw" event={"ID":"46383b3e-ad21-4d92-a14f-e265821d0d7f","Type":"ContainerDied","Data":"ae892886dc4b5d820d8f3dc9ab59d8a2b501a376aac8553bd78f78db35c44733"}
Sep 17 17:18:25.790033 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.789901    2566 scope.go:117] "RemoveContainer" containerID="0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583"
Sep 17 17:18:25.799296 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.799277    2566 scope.go:117] "RemoveContainer" containerID="76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b"
Sep 17 17:18:25.808146 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.808130    2566 scope.go:117] "RemoveContainer" containerID="211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383"
Sep 17 17:18:25.812398 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.812209    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"]
Sep 17 17:18:25.814465 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.814446    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-5sqxw"]
Sep 17 17:18:25.816431 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.816415    2566 scope.go:117] "RemoveContainer" containerID="0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583"
Sep 17 17:18:25.816663 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:18:25.816639    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583\": container with ID starting with 0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583 not found: ID does not exist" containerID="0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583"
Sep 17 17:18:25.816711 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.816672    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583"} err="failed to get container status \"0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583\": rpc error: code = NotFound desc = could not find container \"0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583\": container with ID starting with 0406d4598b1f7c4cfb611193e2038ecd0484e845083736836f9ec8bf04812583 not found: ID does not exist"
Sep 17 17:18:25.816711 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.816689    2566 scope.go:117] "RemoveContainer" containerID="76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b"
Sep 17 17:18:25.816932 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:18:25.816913    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b\": container with ID starting with 76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b not found: ID does not exist" containerID="76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b"
Sep 17 17:18:25.816997 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.816941    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b"} err="failed to get container status \"76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b\": rpc error: code = NotFound desc = could not find container \"76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b\": container with ID starting with 76aae64992d11dd0df1939d32e300f200f59eea303caa7b6cf1cf6292855096b not found: ID does not exist"
Sep 17 17:18:25.816997 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.816964    2566 scope.go:117] "RemoveContainer" containerID="211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383"
Sep 17 17:18:25.817220 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:18:25.817204    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383\": container with ID starting with 211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383 not found: ID does not exist" containerID="211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383"
Sep 17 17:18:25.817285 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:25.817223    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383"} err="failed to get container status \"211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383\": rpc error: code = NotFound desc = could not find container \"211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383\": container with ID starting with 211a3ae5e9d56884595817075645e044c842ebe0ff44f106c8682daceaf6c383 not found: ID does not exist"
Sep 17 17:18:26.708552 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.708513    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:18:26.741887 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.741853    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"]
Sep 17 17:18:26.742325 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742306    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container"
Sep 17 17:18:26.742435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742327    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container"
Sep 17 17:18:26.742435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742347    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="storage-initializer"
Sep 17 17:18:26.742435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742355    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="storage-initializer"
Sep 17 17:18:26.742435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742366    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="storage-initializer"
Sep 17 17:18:26.742435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742374    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="storage-initializer"
Sep 17 17:18:26.742435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742391    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container"
Sep 17 17:18:26.742435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742400    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container"
Sep 17 17:18:26.742435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742421    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kube-rbac-proxy"
Sep 17 17:18:26.742435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742429    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kube-rbac-proxy"
Sep 17 17:18:26.742902 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742440    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kube-rbac-proxy"
Sep 17 17:18:26.742902 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742448    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kube-rbac-proxy"
Sep 17 17:18:26.742902 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742547    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kserve-container"
Sep 17 17:18:26.742902 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742558    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kube-rbac-proxy"
Sep 17 17:18:26.742902 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742567    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c94378-12ba-4972-aec7-4c7bf03a4dac" containerName="kserve-container"
Sep 17 17:18:26.742902 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.742580    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" containerName="kube-rbac-proxy"
Sep 17 17:18:26.746096 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.746076    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.748746 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.748721    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 17:18:26.748746 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.748741    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:18:26.748919 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.748727    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:18:26.748919 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.748891    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 17:18:26.749755 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.749731    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:18:26.749867 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.749759    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:18:26.749867 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.749847    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:18:26.756159 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.756138    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"]
Sep 17 17:18:26.895137 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.895104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.895313 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.895163    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mxds\" (UniqueName: \"kubernetes.io/projected/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-api-access-8mxds\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.895313 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.895186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.895313 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.895202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.895313 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.895223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.910680 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.910656    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:18:26.996413 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.996387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mxds\" (UniqueName: \"kubernetes.io/projected/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-api-access-8mxds\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.996539 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.996419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.996539 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.996437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.996539 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.996485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.996539 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.996535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.997037 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.997017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.997206 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.997189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.997280 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.997242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:26.998881 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:26.998856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:27.003967 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:27.003947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mxds\" (UniqueName: \"kubernetes.io/projected/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-api-access-8mxds\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:27.057549 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:27.057529    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:27.328790 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:27.328763    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"]
Sep 17 17:18:27.330479 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:18:27.330453    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb3586a_6023_45ee_8ed7_2f36e36df8ce.slice/crio-5d1e675278846661c68d7732ca2551a42428d29a12f45a023c5a845633c24a26 WatchSource:0}: Error finding container 5d1e675278846661c68d7732ca2551a42428d29a12f45a023c5a845633c24a26: Status 404 returned error can't find the container with id 5d1e675278846661c68d7732ca2551a42428d29a12f45a023c5a845633c24a26
Sep 17 17:18:27.810151 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:27.810119    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46383b3e-ad21-4d92-a14f-e265821d0d7f" path="/var/lib/kubelet/pods/46383b3e-ad21-4d92-a14f-e265821d0d7f/volumes"
Sep 17 17:18:27.810519 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:27.810466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" event={"ID":"4bb3586a-6023-45ee-8ed7-2f36e36df8ce","Type":"ContainerStarted","Data":"e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933"}
Sep 17 17:18:27.810519 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:27.810512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" event={"ID":"4bb3586a-6023-45ee-8ed7-2f36e36df8ce","Type":"ContainerStarted","Data":"5d1e675278846661c68d7732ca2551a42428d29a12f45a023c5a845633c24a26"}
Sep 17 17:18:31.827522 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:31.827487    2566 generic.go:356] "Generic (PLEG): container finished" podID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerID="e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933" exitCode=0
Sep 17 17:18:31.827947 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:31.827521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" event={"ID":"4bb3586a-6023-45ee-8ed7-2f36e36df8ce","Type":"ContainerDied","Data":"e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933"}
Sep 17 17:18:32.836068 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:32.836036    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" event={"ID":"4bb3586a-6023-45ee-8ed7-2f36e36df8ce","Type":"ContainerStarted","Data":"7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2"}
Sep 17 17:18:32.836465 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:32.836095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" event={"ID":"4bb3586a-6023-45ee-8ed7-2f36e36df8ce","Type":"ContainerStarted","Data":"7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a"}
Sep 17 17:18:32.836465 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:32.836324    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:32.857895 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:32.857848    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" podStartSLOduration=2.874995385 podStartE2EDuration="6.857834129s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.982838744s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:18:32.855918884 +0000 UTC m=+2221.567926154" watchObservedRunningTime="2026-09-17 17:18:32.857834129 +0000 UTC m=+2221.569841396"
Sep 17 17:18:33.839816 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:33.839789    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:39.850938 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:39.850911    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:18:59.931669 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:18:59.931585    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:19:05.133700 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:05.133666    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:19:05.527728 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:05.527688    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:19:09.854800 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:09.854771    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:19:16.786118 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:16.786070    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"]
Sep 17 17:19:16.786563 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:16.786397    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="kserve-container" containerID="cri-o://7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a" gracePeriod=30
Sep 17 17:19:16.786563 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:16.786479    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="kube-rbac-proxy" containerID="cri-o://7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2" gracePeriod=30
Sep 17 17:19:16.808343 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:16.808310    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:19:17.045446 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:17.045364    2566 generic.go:356] "Generic (PLEG): container finished" podID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerID="7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2" exitCode=2
Sep 17 17:19:17.045583 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:17.045436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" event={"ID":"4bb3586a-6023-45ee-8ed7-2f36e36df8ce","Type":"ContainerDied","Data":"7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2"}
Sep 17 17:19:19.845578 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:19.845537    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.59:8643/healthz\": dial tcp 10.133.0.59:8643: connect: connection refused"
Sep 17 17:19:19.851433 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:19.851401    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.59:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 17 17:19:21.952326 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:21.952293    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:19:21.988757 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:21.988722    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"]
Sep 17 17:19:21.991393 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:21.991375    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:21.993728 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:21.993705    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 17:19:21.993999 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:21.993982    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 17:19:22.006141 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.006117    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"]
Sep 17 17:19:22.138348 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.138317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2gx\" (UniqueName: \"kubernetes.io/projected/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-api-access-9c2gx\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.138536 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.138401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd52a20f-55ea-458d-97a0-89471c76cfa2-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.138536 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.138456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd52a20f-55ea-458d-97a0-89471c76cfa2-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.138536 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.138482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.138536 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.138511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.239817 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.239793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2gx\" (UniqueName: \"kubernetes.io/projected/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-api-access-9c2gx\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.239911 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.239847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd52a20f-55ea-458d-97a0-89471c76cfa2-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.239911 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.239882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd52a20f-55ea-458d-97a0-89471c76cfa2-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.239911 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.239897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.240083 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.239917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.240083 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:19:22.240017    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 17:19:22.240207 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:19:22.240087    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd52a20f-55ea-458d-97a0-89471c76cfa2-proxy-tls podName:bd52a20f-55ea-458d-97a0-89471c76cfa2 nodeName:}" failed. No retries permitted until 2026-09-17 17:19:22.740068916 +0000 UTC m=+2271.452076162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd52a20f-55ea-458d-97a0-89471c76cfa2-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-b5942" (UID: "bd52a20f-55ea-458d-97a0-89471c76cfa2") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 17:19:22.240347 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.240330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd52a20f-55ea-458d-97a0-89471c76cfa2-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.240719 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.240698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.240757 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.240698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.248228 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.248210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2gx\" (UniqueName: \"kubernetes.io/projected/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-api-access-9c2gx\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.325605 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.325572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:19:22.423058 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.423037    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:19:22.542480 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.542455    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-rbac-proxy-sar-config\") pod \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") "
Sep 17 17:19:22.542623 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.542502    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-cabundle-cert\") pod \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") "
Sep 17 17:19:22.542623 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.542585    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-proxy-tls\") pod \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") "
Sep 17 17:19:22.542746 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.542701    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mxds\" (UniqueName: \"kubernetes.io/projected/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-api-access-8mxds\") pod \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") "
Sep 17 17:19:22.542815 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.542763    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kserve-provision-location\") pod \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\" (UID: \"4bb3586a-6023-45ee-8ed7-2f36e36df8ce\") "
Sep 17 17:19:22.542815 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.542771    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4bb3586a-6023-45ee-8ed7-2f36e36df8ce" (UID: "4bb3586a-6023-45ee-8ed7-2f36e36df8ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:19:22.542935 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.542898    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4bb3586a-6023-45ee-8ed7-2f36e36df8ce" (UID: "4bb3586a-6023-45ee-8ed7-2f36e36df8ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:19:22.543105 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.543084    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4bb3586a-6023-45ee-8ed7-2f36e36df8ce" (UID: "4bb3586a-6023-45ee-8ed7-2f36e36df8ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:19:22.543186 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.543160    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:19:22.543186 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.543174    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:19:22.543326 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.543186    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:19:22.544705 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.544684    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4bb3586a-6023-45ee-8ed7-2f36e36df8ce" (UID: "4bb3586a-6023-45ee-8ed7-2f36e36df8ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:19:22.544829 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.544812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-api-access-8mxds" (OuterVolumeSpecName: "kube-api-access-8mxds") pod "4bb3586a-6023-45ee-8ed7-2f36e36df8ce" (UID: "4bb3586a-6023-45ee-8ed7-2f36e36df8ce"). InnerVolumeSpecName "kube-api-access-8mxds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:19:22.643525 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.643498    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:19:22.643525 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.643519    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mxds\" (UniqueName: \"kubernetes.io/projected/4bb3586a-6023-45ee-8ed7-2f36e36df8ce-kube-api-access-8mxds\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:19:22.744360 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.744334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd52a20f-55ea-458d-97a0-89471c76cfa2-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.746651 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.746629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd52a20f-55ea-458d-97a0-89471c76cfa2-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-b5942\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:22.901574 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:22.901519    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:23.074562 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.074529    2566 generic.go:356] "Generic (PLEG): container finished" podID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerID="7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a" exitCode=0
Sep 17 17:19:23.074960 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.074581    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"
Sep 17 17:19:23.074960 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.074584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" event={"ID":"4bb3586a-6023-45ee-8ed7-2f36e36df8ce","Type":"ContainerDied","Data":"7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a"}
Sep 17 17:19:23.074960 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.074644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg" event={"ID":"4bb3586a-6023-45ee-8ed7-2f36e36df8ce","Type":"ContainerDied","Data":"5d1e675278846661c68d7732ca2551a42428d29a12f45a023c5a845633c24a26"}
Sep 17 17:19:23.074960 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.074671    2566 scope.go:117] "RemoveContainer" containerID="7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2"
Sep 17 17:19:23.083001 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.082969    2566 scope.go:117] "RemoveContainer" containerID="7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a"
Sep 17 17:19:23.090398 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.090381    2566 scope.go:117] "RemoveContainer" containerID="e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933"
Sep 17 17:19:23.096076 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.096056    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"]
Sep 17 17:19:23.098001 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.097989    2566 scope.go:117] "RemoveContainer" containerID="7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2"
Sep 17 17:19:23.098310 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:19:23.098237    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2\": container with ID starting with 7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2 not found: ID does not exist" containerID="7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2"
Sep 17 17:19:23.098310 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.098277    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2"} err="failed to get container status \"7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2\": rpc error: code = NotFound desc = could not find container \"7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2\": container with ID starting with 7695e67420e038dead378261f2fa3035564041238a76eb48e476cb7051e079c2 not found: ID does not exist"
Sep 17 17:19:23.098310 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.098293    2566 scope.go:117] "RemoveContainer" containerID="7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a"
Sep 17 17:19:23.098534 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:19:23.098520    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a\": container with ID starting with 7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a not found: ID does not exist" containerID="7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a"
Sep 17 17:19:23.098574 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.098540    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a"} err="failed to get container status \"7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a\": rpc error: code = NotFound desc = could not find container \"7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a\": container with ID starting with 7632b542a525935e0001f32bb4d13ce47da253f69f52f0bd05ff23d3d0c9916a not found: ID does not exist"
Sep 17 17:19:23.098574 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.098553    2566 scope.go:117] "RemoveContainer" containerID="e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933"
Sep 17 17:19:23.098790 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:19:23.098774    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933\": container with ID starting with e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933 not found: ID does not exist" containerID="e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933"
Sep 17 17:19:23.098841 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.098794    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933"} err="failed to get container status \"e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933\": rpc error: code = NotFound desc = could not find container \"e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933\": container with ID starting with e406a52eb1d5da330569640e73ea25c0e182acba86020108d6fa49051f94b933 not found: ID does not exist"
Sep 17 17:19:23.101472 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.101453    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-7dnsg"]
Sep 17 17:19:23.172388 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.172336    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"]
Sep 17 17:19:23.175157 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:19:23.175136    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd52a20f_55ea_458d_97a0_89471c76cfa2.slice/crio-a6c35c8756165d0053903c0868b14caf86b19314b8224a0243041241f645f481 WatchSource:0}: Error finding container a6c35c8756165d0053903c0868b14caf86b19314b8224a0243041241f645f481: Status 404 returned error can't find the container with id a6c35c8756165d0053903c0868b14caf86b19314b8224a0243041241f645f481
Sep 17 17:19:23.809622 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:23.809588    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" path="/var/lib/kubelet/pods/4bb3586a-6023-45ee-8ed7-2f36e36df8ce/volumes"
Sep 17 17:19:24.081941 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:24.081855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" event={"ID":"bd52a20f-55ea-458d-97a0-89471c76cfa2","Type":"ContainerStarted","Data":"55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a"}
Sep 17 17:19:24.081941 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:24.081907    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" event={"ID":"bd52a20f-55ea-458d-97a0-89471c76cfa2","Type":"ContainerStarted","Data":"a6c35c8756165d0053903c0868b14caf86b19314b8224a0243041241f645f481"}
Sep 17 17:19:27.097184 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:27.097155    2566 generic.go:356] "Generic (PLEG): container finished" podID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerID="55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a" exitCode=0
Sep 17 17:19:27.097501 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:27.097231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" event={"ID":"bd52a20f-55ea-458d-97a0-89471c76cfa2","Type":"ContainerDied","Data":"55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a"}
Sep 17 17:19:28.104674 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:28.104639    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" event={"ID":"bd52a20f-55ea-458d-97a0-89471c76cfa2","Type":"ContainerStarted","Data":"5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1"}
Sep 17 17:19:28.105048 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:28.104703    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" event={"ID":"bd52a20f-55ea-458d-97a0-89471c76cfa2","Type":"ContainerStarted","Data":"1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae"}
Sep 17 17:19:28.105048 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:28.104870    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:28.128457 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:28.128400    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" podStartSLOduration=3.3740319899999998 podStartE2EDuration="7.128382973s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.754350983s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:19:28.124117166 +0000 UTC m=+2276.836124427" watchObservedRunningTime="2026-09-17 17:19:28.128382973 +0000 UTC m=+2276.840390243"
Sep 17 17:19:29.109201 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:29.109173    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:19:35.119016 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:19:35.118980    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:20:05.123344 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:05.123311    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:20:12.034006 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:12.033973    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"]
Sep 17 17:20:12.034531 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:12.034230    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="kserve-container" containerID="cri-o://1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae" gracePeriod=30
Sep 17 17:20:12.034531 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:12.034290    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="kube-rbac-proxy" containerID="cri-o://5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1" gracePeriod=30
Sep 17 17:20:12.066943 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:12.066899    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:20:12.308796 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:12.308715    2566 generic.go:356] "Generic (PLEG): container finished" podID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerID="5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1" exitCode=2
Sep 17 17:20:12.308796 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:12.308781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" event={"ID":"bd52a20f-55ea-458d-97a0-89471c76cfa2","Type":"ContainerDied","Data":"5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1"}
Sep 17 17:20:15.114470 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:15.114432    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Sep 17 17:20:15.119849 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:15.119824    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.60:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 17 17:20:17.197362 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.197321    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:20:17.281370 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.281346    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:20:17.334187 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.334158    2566 generic.go:356] "Generic (PLEG): container finished" podID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerID="1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae" exitCode=0
Sep 17 17:20:17.334365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.334214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" event={"ID":"bd52a20f-55ea-458d-97a0-89471c76cfa2","Type":"ContainerDied","Data":"1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae"}
Sep 17 17:20:17.334365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.334223    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"
Sep 17 17:20:17.334365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.334294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942" event={"ID":"bd52a20f-55ea-458d-97a0-89471c76cfa2","Type":"ContainerDied","Data":"a6c35c8756165d0053903c0868b14caf86b19314b8224a0243041241f645f481"}
Sep 17 17:20:17.334365 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.334335    2566 scope.go:117] "RemoveContainer" containerID="5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1"
Sep 17 17:20:17.342237 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.342220    2566 scope.go:117] "RemoveContainer" containerID="1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae"
Sep 17 17:20:17.349696 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.349679    2566 scope.go:117] "RemoveContainer" containerID="55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a"
Sep 17 17:20:17.356736 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.356710    2566 scope.go:117] "RemoveContainer" containerID="5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1"
Sep 17 17:20:17.356991 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:20:17.356972    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1\": container with ID starting with 5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1 not found: ID does not exist" containerID="5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1"
Sep 17 17:20:17.357064 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.357003    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1"} err="failed to get container status \"5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1\": rpc error: code = NotFound desc = could not find container \"5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1\": container with ID starting with 5acb6d56453ef6033873b8aa2118fde89c7398dc9b7a41c65144388a96c5d2c1 not found: ID does not exist"
Sep 17 17:20:17.357064 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.357025    2566 scope.go:117] "RemoveContainer" containerID="1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae"
Sep 17 17:20:17.357293 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:20:17.357275    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae\": container with ID starting with 1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae not found: ID does not exist" containerID="1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae"
Sep 17 17:20:17.357345 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.357300    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae"} err="failed to get container status \"1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae\": rpc error: code = NotFound desc = could not find container \"1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae\": container with ID starting with 1348fdcf0dad0cd3f182f594c036fa080fe2a1b3a53ab6f17f5cb853708b02ae not found: ID does not exist"
Sep 17 17:20:17.357345 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.357315    2566 scope.go:117] "RemoveContainer" containerID="55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a"
Sep 17 17:20:17.357553 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:20:17.357534    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a\": container with ID starting with 55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a not found: ID does not exist" containerID="55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a"
Sep 17 17:20:17.357614 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.357561    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a"} err="failed to get container status \"55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a\": rpc error: code = NotFound desc = could not find container \"55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a\": container with ID starting with 55f8dda657e3d55fc62ad50b1c27eaeea1fc8b83977dc8bf95c45c3c427e914a not found: ID does not exist"
Sep 17 17:20:17.370304 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.370288    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-rbac-proxy-sar-config\") pod \"bd52a20f-55ea-458d-97a0-89471c76cfa2\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") "
Sep 17 17:20:17.370385 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.370330    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd52a20f-55ea-458d-97a0-89471c76cfa2-proxy-tls\") pod \"bd52a20f-55ea-458d-97a0-89471c76cfa2\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") "
Sep 17 17:20:17.370385 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.370358    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c2gx\" (UniqueName: \"kubernetes.io/projected/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-api-access-9c2gx\") pod \"bd52a20f-55ea-458d-97a0-89471c76cfa2\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") "
Sep 17 17:20:17.370486 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.370390    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-cabundle-cert\") pod \"bd52a20f-55ea-458d-97a0-89471c76cfa2\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") "
Sep 17 17:20:17.370486 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.370423    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd52a20f-55ea-458d-97a0-89471c76cfa2-kserve-provision-location\") pod \"bd52a20f-55ea-458d-97a0-89471c76cfa2\" (UID: \"bd52a20f-55ea-458d-97a0-89471c76cfa2\") "
Sep 17 17:20:17.370659 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.370589    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd52a20f-55ea-458d-97a0-89471c76cfa2" (UID: "bd52a20f-55ea-458d-97a0-89471c76cfa2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:20:17.370785 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.370760    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd52a20f-55ea-458d-97a0-89471c76cfa2" (UID: "bd52a20f-55ea-458d-97a0-89471c76cfa2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:20:17.370848 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.370800    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd52a20f-55ea-458d-97a0-89471c76cfa2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd52a20f-55ea-458d-97a0-89471c76cfa2" (UID: "bd52a20f-55ea-458d-97a0-89471c76cfa2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:20:17.372460 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.372438    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd52a20f-55ea-458d-97a0-89471c76cfa2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd52a20f-55ea-458d-97a0-89471c76cfa2" (UID: "bd52a20f-55ea-458d-97a0-89471c76cfa2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:20:17.372545 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.372441    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-api-access-9c2gx" (OuterVolumeSpecName: "kube-api-access-9c2gx") pod "bd52a20f-55ea-458d-97a0-89471c76cfa2" (UID: "bd52a20f-55ea-458d-97a0-89471c76cfa2"). InnerVolumeSpecName "kube-api-access-9c2gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:20:17.420151 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.420101    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:20:17.471338 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.471315    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c2gx\" (UniqueName: \"kubernetes.io/projected/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-api-access-9c2gx\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:20:17.471338 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.471336    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:20:17.471455 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.471345    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd52a20f-55ea-458d-97a0-89471c76cfa2-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:20:17.471455 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.471354    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd52a20f-55ea-458d-97a0-89471c76cfa2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:20:17.471455 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.471363    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd52a20f-55ea-458d-97a0-89471c76cfa2-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:20:17.655722 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.655692    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"]
Sep 17 17:20:17.659447 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.659425    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-b5942"]
Sep 17 17:20:17.809675 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:17.809649    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" path="/var/lib/kubelet/pods/bd52a20f-55ea-458d-97a0-89471c76cfa2/volumes"
Sep 17 17:20:56.995421 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:56.995384    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:20:57.121242 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:20:57.121205    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:21:02.051079 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.051011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:21:02.089950 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.089918    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"]
Sep 17 17:21:02.090289 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090274    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="kube-rbac-proxy"
Sep 17 17:21:02.090362 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090292    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="kube-rbac-proxy"
Sep 17 17:21:02.090362 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090303    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="kube-rbac-proxy"
Sep 17 17:21:02.090362 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090312    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="kube-rbac-proxy"
Sep 17 17:21:02.090362 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090323    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="storage-initializer"
Sep 17 17:21:02.090362 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090332    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="storage-initializer"
Sep 17 17:21:02.090362 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090345    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="kserve-container"
Sep 17 17:21:02.090362 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090351    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="kserve-container"
Sep 17 17:21:02.090362 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090358    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="storage-initializer"
Sep 17 17:21:02.090362 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090363    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="storage-initializer"
Sep 17 17:21:02.090672 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090377    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="kserve-container"
Sep 17 17:21:02.090672 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090382    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="kserve-container"
Sep 17 17:21:02.090672 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090450    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="kserve-container"
Sep 17 17:21:02.090672 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090459    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="kube-rbac-proxy"
Sep 17 17:21:02.090672 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090467    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bb3586a-6023-45ee-8ed7-2f36e36df8ce" containerName="kube-rbac-proxy"
Sep 17 17:21:02.090672 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.090472    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd52a20f-55ea-458d-97a0-89471c76cfa2" containerName="kserve-container"
Sep 17 17:21:02.093719 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.093702    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.096030 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.095997    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 17:21:02.096030 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.096027    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 17 17:21:02.096223 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.096083    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:21:02.096289 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.096275    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:21:02.096348 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.096326    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:21:02.096982 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.096965    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:21:02.097045 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.096965    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:21:02.105656 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.105636    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"]
Sep 17 17:21:02.176132 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.176094    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:21:02.197237 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.197212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.197359 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.197267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmb8q\" (UniqueName: \"kubernetes.io/projected/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-api-access-kmb8q\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.197359 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.197339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.197449 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.197400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.197487 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.197449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.298354 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.298306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.298560 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.298374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.298560 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.298419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.298560 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.298445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmb8q\" (UniqueName: \"kubernetes.io/projected/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-api-access-kmb8q\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.298560 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.298476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.298876 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.298855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.299296 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.299272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.299349 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.299279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.300812 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.300789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.309587 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.309533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmb8q\" (UniqueName: \"kubernetes.io/projected/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-api-access-kmb8q\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.404115 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.404075    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:02.675793 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:02.675767    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"]
Sep 17 17:21:02.680931 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:21:02.680903    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a094e3b_36e2_4da5_b699_e7f44fd63ec4.slice/crio-4bcc6c290e55d8e6be12741709b3ee170648822370f7deb5b98ef6ee0d254c13 WatchSource:0}: Error finding container 4bcc6c290e55d8e6be12741709b3ee170648822370f7deb5b98ef6ee0d254c13: Status 404 returned error can't find the container with id 4bcc6c290e55d8e6be12741709b3ee170648822370f7deb5b98ef6ee0d254c13
Sep 17 17:21:03.551864 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:03.551826    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6" event={"ID":"3a094e3b-36e2-4da5-b699-e7f44fd63ec4","Type":"ContainerStarted","Data":"23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b"}
Sep 17 17:21:03.552228 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:03.551909    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6" event={"ID":"3a094e3b-36e2-4da5-b699-e7f44fd63ec4","Type":"ContainerStarted","Data":"4bcc6c290e55d8e6be12741709b3ee170648822370f7deb5b98ef6ee0d254c13"}
Sep 17 17:21:07.570764 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:07.570720    2566 generic.go:356] "Generic (PLEG): container finished" podID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerID="23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b" exitCode=0
Sep 17 17:21:07.571158 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:07.570791    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6" event={"ID":"3a094e3b-36e2-4da5-b699-e7f44fd63ec4","Type":"ContainerDied","Data":"23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b"}
Sep 17 17:21:08.133943 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:08.133910    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:21:08.578589 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:08.578554    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6" event={"ID":"3a094e3b-36e2-4da5-b699-e7f44fd63ec4","Type":"ContainerStarted","Data":"3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557"}
Sep 17 17:21:08.578945 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:08.578619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6" event={"ID":"3a094e3b-36e2-4da5-b699-e7f44fd63ec4","Type":"ContainerStarted","Data":"148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280"}
Sep 17 17:21:08.578945 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:08.578810    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:08.578945 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:08.578864    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:08.598509 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:08.598463    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6" podStartSLOduration=2.696523247 podStartE2EDuration="6.598449484s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.901926237s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:21:08.597666152 +0000 UTC m=+2377.309673419" watchObservedRunningTime="2026-09-17 17:21:08.598449484 +0000 UTC m=+2377.310456751"
Sep 17 17:21:14.589182 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:14.589148    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:44.594019 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:44.593972    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:52.175817 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:52.175780    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"]
Sep 17 17:21:52.176395 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:52.176020    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6" podUID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerName="kserve-container" containerID="cri-o://148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280" gracePeriod=30
Sep 17 17:21:52.176395 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:52.176047    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6" podUID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerName="kube-rbac-proxy" containerID="cri-o://3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557" gracePeriod=30
Sep 17 17:21:52.191331 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:52.191301    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:21:52.794079 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:52.794046    2566 generic.go:356] "Generic (PLEG): container finished" podID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerID="3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557" exitCode=2
Sep 17 17:21:52.794317 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:52.794129    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6" event={"ID":"3a094e3b-36e2-4da5-b699-e7f44fd63ec4","Type":"ContainerDied","Data":"3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557"}
Sep 17 17:21:53.122181 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.122157    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:53.207363 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.207335    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmb8q\" (UniqueName: \"kubernetes.io/projected/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-api-access-kmb8q\") pod \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") "
Sep 17 17:21:53.207363 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.207365    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-proxy-tls\") pod \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") "
Sep 17 17:21:53.207751 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.207395    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-rbac-proxy-sar-config\") pod \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") "
Sep 17 17:21:53.207751 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.207471    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kserve-provision-location\") pod \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") "
Sep 17 17:21:53.207751 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.207551    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-cabundle-cert\") pod \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\" (UID: \"3a094e3b-36e2-4da5-b699-e7f44fd63ec4\") "
Sep 17 17:21:53.207924 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.207790    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a094e3b-36e2-4da5-b699-e7f44fd63ec4" (UID: "3a094e3b-36e2-4da5-b699-e7f44fd63ec4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:21:53.207924 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.207833    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a094e3b-36e2-4da5-b699-e7f44fd63ec4" (UID: "3a094e3b-36e2-4da5-b699-e7f44fd63ec4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:21:53.208024 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.207960    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a094e3b-36e2-4da5-b699-e7f44fd63ec4" (UID: "3a094e3b-36e2-4da5-b699-e7f44fd63ec4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:21:53.209395 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.209375    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-api-access-kmb8q" (OuterVolumeSpecName: "kube-api-access-kmb8q") pod "3a094e3b-36e2-4da5-b699-e7f44fd63ec4" (UID: "3a094e3b-36e2-4da5-b699-e7f44fd63ec4"). InnerVolumeSpecName "kube-api-access-kmb8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:21:53.209473 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.209444    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a094e3b-36e2-4da5-b699-e7f44fd63ec4" (UID: "3a094e3b-36e2-4da5-b699-e7f44fd63ec4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:21:53.309106 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.309083    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:21:53.309106 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.309102    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:21:53.309240 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.309114    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmb8q\" (UniqueName: \"kubernetes.io/projected/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-api-access-kmb8q\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:21:53.309240 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.309122    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:21:53.309240 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.309130    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a094e3b-36e2-4da5-b699-e7f44fd63ec4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:21:53.800488 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.800453    2566 generic.go:356] "Generic (PLEG): container finished" podID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerID="148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280" exitCode=0
Sep 17 17:21:53.800678 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.800509    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"
Sep 17 17:21:53.800678 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.800540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6" event={"ID":"3a094e3b-36e2-4da5-b699-e7f44fd63ec4","Type":"ContainerDied","Data":"148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280"}
Sep 17 17:21:53.800678 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.800618    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6" event={"ID":"3a094e3b-36e2-4da5-b699-e7f44fd63ec4","Type":"ContainerDied","Data":"4bcc6c290e55d8e6be12741709b3ee170648822370f7deb5b98ef6ee0d254c13"}
Sep 17 17:21:53.800678 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.800655    2566 scope.go:117] "RemoveContainer" containerID="3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557"
Sep 17 17:21:53.810137 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.810125    2566 scope.go:117] "RemoveContainer" containerID="148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280"
Sep 17 17:21:53.817984 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.817968    2566 scope.go:117] "RemoveContainer" containerID="23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b"
Sep 17 17:21:53.824103 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.824081    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"]
Sep 17 17:21:53.826186 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.826167    2566 scope.go:117] "RemoveContainer" containerID="3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557"
Sep 17 17:21:53.826715 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:21:53.826584    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557\": container with ID starting with 3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557 not found: ID does not exist" containerID="3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557"
Sep 17 17:21:53.826715 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.826619    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557"} err="failed to get container status \"3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557\": rpc error: code = NotFound desc = could not find container \"3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557\": container with ID starting with 3996ccd1948b9c746113a6f725eb2363c2bd7ebb04714a4a805c53b0d7eba557 not found: ID does not exist"
Sep 17 17:21:53.826715 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.826643    2566 scope.go:117] "RemoveContainer" containerID="148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280"
Sep 17 17:21:53.826963 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:21:53.826942    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280\": container with ID starting with 148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280 not found: ID does not exist" containerID="148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280"
Sep 17 17:21:53.827029 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.826972    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280"} err="failed to get container status \"148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280\": rpc error: code = NotFound desc = could not find container \"148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280\": container with ID starting with 148b1e661ae677a47c5696c17796785b3cba45f929cdfb94f294405f43edd280 not found: ID does not exist"
Sep 17 17:21:53.827029 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.826990    2566 scope.go:117] "RemoveContainer" containerID="23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b"
Sep 17 17:21:53.827242 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:21:53.827220    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b\": container with ID starting with 23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b not found: ID does not exist" containerID="23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b"
Sep 17 17:21:53.827340 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.827248    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b"} err="failed to get container status \"23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b\": rpc error: code = NotFound desc = could not find container \"23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b\": container with ID starting with 23ac6ea8aa7afef8e63d5aec7bf6cd9a93e46a234f5874e1c80c36ac5c4fb24b not found: ID does not exist"
Sep 17 17:21:53.829027 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:53.829006    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-p58p6"]
Sep 17 17:21:55.816687 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:55.816651    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" path="/var/lib/kubelet/pods/3a094e3b-36e2-4da5-b699-e7f44fd63ec4/volumes"
Sep 17 17:21:57.314211 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.314176    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:21:57.345801 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.345768    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"]
Sep 17 17:21:57.346181 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.346165    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerName="kube-rbac-proxy"
Sep 17 17:21:57.346311 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.346183    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerName="kube-rbac-proxy"
Sep 17 17:21:57.346311 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.346212    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerName="storage-initializer"
Sep 17 17:21:57.346311 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.346221    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerName="storage-initializer"
Sep 17 17:21:57.346311 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.346235    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerName="kserve-container"
Sep 17 17:21:57.346311 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.346244    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerName="kserve-container"
Sep 17 17:21:57.346587 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.346352    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerName="kserve-container"
Sep 17 17:21:57.346587 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.346366    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a094e3b-36e2-4da5-b699-e7f44fd63ec4" containerName="kube-rbac-proxy"
Sep 17 17:21:57.351348 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.351327    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.353975 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.353954    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:21:57.354351 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.354318    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 17 17:21:57.354447 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.354365    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 17:21:57.354665 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.354641    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:21:57.354758 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.354641    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:21:57.355209 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.355193    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:21:57.355209 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.355208    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:21:57.360317 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.360295    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"]
Sep 17 17:21:57.440757 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.440731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvfw\" (UniqueName: \"kubernetes.io/projected/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-api-access-cdvfw\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.440885 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.440840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.440885 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.440865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.440989 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.440929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.440989 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.440967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.468829 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.468800    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:21:57.541911 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.541883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.541911 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.541911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.542075 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.541942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.542075 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.541966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.542075 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.542008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvfw\" (UniqueName: \"kubernetes.io/projected/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-api-access-cdvfw\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.542360 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.542338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.542650 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.542629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.542725 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.542708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.544434 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.544417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.550087 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.550056    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvfw\" (UniqueName: \"kubernetes.io/projected/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-api-access-cdvfw\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.663401 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.663347    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:21:57.748548 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.748520    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:21:57.944690 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:57.944662    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"]
Sep 17 17:21:57.946468 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:21:57.946446    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe419ee_30a0_4cac_bebd_affd0b1a44f3.slice/crio-32e17f995a77736121b8dbfd9ff38a21144fea79bd57caf54c4e2b573699ce68 WatchSource:0}: Error finding container 32e17f995a77736121b8dbfd9ff38a21144fea79bd57caf54c4e2b573699ce68: Status 404 returned error can't find the container with id 32e17f995a77736121b8dbfd9ff38a21144fea79bd57caf54c4e2b573699ce68
Sep 17 17:21:58.832624 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:58.832583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" event={"ID":"6fe419ee-30a0-4cac-bebd-affd0b1a44f3","Type":"ContainerStarted","Data":"160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef"}
Sep 17 17:21:58.833016 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:21:58.832652    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" event={"ID":"6fe419ee-30a0-4cac-bebd-affd0b1a44f3","Type":"ContainerStarted","Data":"32e17f995a77736121b8dbfd9ff38a21144fea79bd57caf54c4e2b573699ce68"}
Sep 17 17:22:01.847268 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:01.847220    2566 generic.go:356] "Generic (PLEG): container finished" podID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerID="160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef" exitCode=0
Sep 17 17:22:01.847608 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:01.847291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" event={"ID":"6fe419ee-30a0-4cac-bebd-affd0b1a44f3","Type":"ContainerDied","Data":"160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef"}
Sep 17 17:22:01.848591 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:01.848576    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:22:02.871313 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:02.871279    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:22:02.912765 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:02.912735    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"]
Sep 17 17:22:02.916342 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:02.916325    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:02.918944 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:02.918908    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 17:22:02.919072 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:02.918909    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 17 17:22:02.927871 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:02.927848    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"]
Sep 17 17:22:02.986507 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:02.986477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:02.986677 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:02.986515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:02.986677 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:02.986545    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75efa128-178a-42c3-853c-6537ab7e5dbc-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:02.986677 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:02.986599    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75efa128-178a-42c3-853c-6537ab7e5dbc-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:02.986677 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:02.986644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cndt4\" (UniqueName: \"kubernetes.io/projected/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-api-access-cndt4\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.087824 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.087789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cndt4\" (UniqueName: \"kubernetes.io/projected/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-api-access-cndt4\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.088004 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.087885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.088004 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.087917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.088126 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.088058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75efa128-178a-42c3-853c-6537ab7e5dbc-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.088126 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.088102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75efa128-178a-42c3-853c-6537ab7e5dbc-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.088287 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:22:03.088248    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 17 17:22:03.088359 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:22:03.088339    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75efa128-178a-42c3-853c-6537ab7e5dbc-proxy-tls podName:75efa128-178a-42c3-853c-6537ab7e5dbc nodeName:}" failed. No retries permitted until 2026-09-17 17:22:03.588319121 +0000 UTC m=+2432.300326369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/75efa128-178a-42c3-853c-6537ab7e5dbc-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" (UID: "75efa128-178a-42c3-853c-6537ab7e5dbc") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 17 17:22:03.088707 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.088663    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75efa128-178a-42c3-853c-6537ab7e5dbc-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.088846 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.088737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.088846 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.088759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.096602 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.096577    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cndt4\" (UniqueName: \"kubernetes.io/projected/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-api-access-cndt4\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.492294 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.492228    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:22:03.592717 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.592649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75efa128-178a-42c3-853c-6537ab7e5dbc-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.595557 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.595530    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75efa128-178a-42c3-853c-6537ab7e5dbc-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:03.830656 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:03.830581    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:04.115635 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:04.115607    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"]
Sep 17 17:22:04.117579 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:22:04.117552    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75efa128_178a_42c3_853c_6537ab7e5dbc.slice/crio-e1a79dc350a810289c3bb06be95c919955927c86261b1880bd1ae46da634b4f0 WatchSource:0}: Error finding container e1a79dc350a810289c3bb06be95c919955927c86261b1880bd1ae46da634b4f0: Status 404 returned error can't find the container with id e1a79dc350a810289c3bb06be95c919955927c86261b1880bd1ae46da634b4f0
Sep 17 17:22:04.867108 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:04.867060    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" event={"ID":"75efa128-178a-42c3-853c-6537ab7e5dbc","Type":"ContainerStarted","Data":"3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c"}
Sep 17 17:22:04.867324 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:04.867144    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" event={"ID":"75efa128-178a-42c3-853c-6537ab7e5dbc","Type":"ContainerStarted","Data":"e1a79dc350a810289c3bb06be95c919955927c86261b1880bd1ae46da634b4f0"}
Sep 17 17:22:08.890187 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:08.890152    2566 generic.go:356] "Generic (PLEG): container finished" podID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerID="3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c" exitCode=0
Sep 17 17:22:08.890617 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:08.890227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" event={"ID":"75efa128-178a-42c3-853c-6537ab7e5dbc","Type":"ContainerDied","Data":"3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c"}
Sep 17 17:22:08.892661 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:08.892637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" event={"ID":"6fe419ee-30a0-4cac-bebd-affd0b1a44f3","Type":"ContainerStarted","Data":"5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01"}
Sep 17 17:22:08.892763 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:08.892687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" event={"ID":"6fe419ee-30a0-4cac-bebd-affd0b1a44f3","Type":"ContainerStarted","Data":"a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d"}
Sep 17 17:22:08.892882 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:08.892866    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:22:08.926267 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:08.926173    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" podStartSLOduration=1.341428315 podStartE2EDuration="11.926157607s" totalImagesPullingTime="6.826493303s" totalInitContainerRuntime="3.758235989s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:22:08.924955611 +0000 UTC m=+2437.636962915" watchObservedRunningTime="2026-09-17 17:22:08.926157607 +0000 UTC m=+2437.638164875"
Sep 17 17:22:09.899588 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:09.899549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" event={"ID":"75efa128-178a-42c3-853c-6537ab7e5dbc","Type":"ContainerStarted","Data":"10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec"}
Sep 17 17:22:09.899997 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:09.899620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" event={"ID":"75efa128-178a-42c3-853c-6537ab7e5dbc","Type":"ContainerStarted","Data":"757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3"}
Sep 17 17:22:09.899997 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:09.899863    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:22:09.900126 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:09.900092    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:09.900308 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:09.900277    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:09.901453 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:09.901428    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 17 17:22:09.920653 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:09.920607    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" podStartSLOduration=3.694612255 podStartE2EDuration="7.920594914s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.225982659s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:22:09.919269516 +0000 UTC m=+2438.631276773" watchObservedRunningTime="2026-09-17 17:22:09.920594914 +0000 UTC m=+2438.632602183"
Sep 17 17:22:10.905998 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:10.905954    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 17 17:22:15.910922 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:15.910887    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:22:15.911609 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:15.911585    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:22:15.912130 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:15.912098    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 17 17:22:25.912969 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:25.912942    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:22:27.441623 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:27.441590    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"]
Sep 17 17:22:27.442009 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:27.441822    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="kserve-container" containerID="cri-o://a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d" gracePeriod=30
Sep 17 17:22:27.442009 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:27.441842    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="kube-rbac-proxy" containerID="cri-o://5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01" gracePeriod=30
Sep 17 17:22:27.452144 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:27.452122    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:22:27.994103 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:27.994075    2566 generic.go:356] "Generic (PLEG): container finished" podID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerID="5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01" exitCode=2
Sep 17 17:22:27.994230 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:27.994127    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" event={"ID":"6fe419ee-30a0-4cac-bebd-affd0b1a44f3","Type":"ContainerDied","Data":"5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01"}
Sep 17 17:22:28.178239 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.178217    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:22:28.311309 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.311286    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kserve-provision-location\") pod \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") "
Sep 17 17:22:28.311453 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.311330    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-rbac-proxy-sar-config\") pod \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") "
Sep 17 17:22:28.311453 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.311370    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdvfw\" (UniqueName: \"kubernetes.io/projected/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-api-access-cdvfw\") pod \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") "
Sep 17 17:22:28.311453 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.311386    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-proxy-tls\") pod \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") "
Sep 17 17:22:28.311453 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.311416    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-cabundle-cert\") pod \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\" (UID: \"6fe419ee-30a0-4cac-bebd-affd0b1a44f3\") "
Sep 17 17:22:28.311674 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.311652    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6fe419ee-30a0-4cac-bebd-affd0b1a44f3" (UID: "6fe419ee-30a0-4cac-bebd-affd0b1a44f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:22:28.311772 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.311752    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6fe419ee-30a0-4cac-bebd-affd0b1a44f3" (UID: "6fe419ee-30a0-4cac-bebd-affd0b1a44f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:22:28.311868 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.311845    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6fe419ee-30a0-4cac-bebd-affd0b1a44f3" (UID: "6fe419ee-30a0-4cac-bebd-affd0b1a44f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:22:28.313493 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.313468    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6fe419ee-30a0-4cac-bebd-affd0b1a44f3" (UID: "6fe419ee-30a0-4cac-bebd-affd0b1a44f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:22:28.313493 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.313487    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-api-access-cdvfw" (OuterVolumeSpecName: "kube-api-access-cdvfw") pod "6fe419ee-30a0-4cac-bebd-affd0b1a44f3" (UID: "6fe419ee-30a0-4cac-bebd-affd0b1a44f3"). InnerVolumeSpecName "kube-api-access-cdvfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:22:28.412732 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.412694    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:22:28.412732 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.412728    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:22:28.412879 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.412745    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:22:28.412879 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.412759    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdvfw\" (UniqueName: \"kubernetes.io/projected/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-kube-api-access-cdvfw\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:22:28.412879 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:28.412772    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe419ee-30a0-4cac-bebd-affd0b1a44f3-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:22:29.001150 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.001112    2566 generic.go:356] "Generic (PLEG): container finished" podID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerID="a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d" exitCode=0
Sep 17 17:22:29.001547 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.001172    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" event={"ID":"6fe419ee-30a0-4cac-bebd-affd0b1a44f3","Type":"ContainerDied","Data":"a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d"}
Sep 17 17:22:29.001547 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.001181    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"
Sep 17 17:22:29.001547 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.001289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz" event={"ID":"6fe419ee-30a0-4cac-bebd-affd0b1a44f3","Type":"ContainerDied","Data":"32e17f995a77736121b8dbfd9ff38a21144fea79bd57caf54c4e2b573699ce68"}
Sep 17 17:22:29.001547 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.001331    2566 scope.go:117] "RemoveContainer" containerID="5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01"
Sep 17 17:22:29.011034 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.011018    2566 scope.go:117] "RemoveContainer" containerID="a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d"
Sep 17 17:22:29.018548 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.018527    2566 scope.go:117] "RemoveContainer" containerID="160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef"
Sep 17 17:22:29.024928 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.024908    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"]
Sep 17 17:22:29.029758 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.029734    2566 scope.go:117] "RemoveContainer" containerID="5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01"
Sep 17 17:22:29.030207 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:22:29.030183    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01\": container with ID starting with 5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01 not found: ID does not exist" containerID="5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01"
Sep 17 17:22:29.030332 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.030212    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01"} err="failed to get container status \"5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01\": rpc error: code = NotFound desc = could not find container \"5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01\": container with ID starting with 5b741c87752c3b5082ccc60aab280dc269d606703a6835f5b926174451e90f01 not found: ID does not exist"
Sep 17 17:22:29.030332 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.030228    2566 scope.go:117] "RemoveContainer" containerID="a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d"
Sep 17 17:22:29.030559 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:22:29.030539    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d\": container with ID starting with a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d not found: ID does not exist" containerID="a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d"
Sep 17 17:22:29.030636 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.030570    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d"} err="failed to get container status \"a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d\": rpc error: code = NotFound desc = could not find container \"a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d\": container with ID starting with a8914c6052387e8e3566278624585808eec5cb9b4fa09182e362d9eaa7bd442d not found: ID does not exist"
Sep 17 17:22:29.030636 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.030595    2566 scope.go:117] "RemoveContainer" containerID="160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef"
Sep 17 17:22:29.030905 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:22:29.030887    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef\": container with ID starting with 160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef not found: ID does not exist" containerID="160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef"
Sep 17 17:22:29.030968 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.030909    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef"} err="failed to get container status \"160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef\": rpc error: code = NotFound desc = could not find container \"160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef\": container with ID starting with 160b18c199edc05dbdab2d1d53cd222816c72ea8bc4a8febc9fe82d0c00220ef not found: ID does not exist"
Sep 17 17:22:29.031905 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.031886    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-stkcz"]
Sep 17 17:22:29.810076 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:29.810041    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" path="/var/lib/kubelet/pods/6fe419ee-30a0-4cac-bebd-affd0b1a44f3/volumes"
Sep 17 17:22:32.609374 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.609340    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:22:32.659938 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.659908    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"]
Sep 17 17:22:32.660239 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.660227    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="kserve-container"
Sep 17 17:22:32.660313 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.660240    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="kserve-container"
Sep 17 17:22:32.660313 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.660284    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="storage-initializer"
Sep 17 17:22:32.660313 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.660295    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="storage-initializer"
Sep 17 17:22:32.660313 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.660303    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="kube-rbac-proxy"
Sep 17 17:22:32.660313 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.660308    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="kube-rbac-proxy"
Sep 17 17:22:32.660485 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.660370    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="kserve-container"
Sep 17 17:22:32.660485 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.660378    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe419ee-30a0-4cac-bebd-affd0b1a44f3" containerName="kube-rbac-proxy"
Sep 17 17:22:32.664856 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.664839    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.667827 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.667807    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 17 17:22:32.668084 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.668068    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 17:22:32.679410 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.679389    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"]
Sep 17 17:22:32.745920 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.745892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.746037 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.745924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c211eb-deec-4c6b-a553-bbf965bbf480-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.746037 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.745959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45c211eb-deec-4c6b-a553-bbf965bbf480-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.746112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.746033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgwt\" (UniqueName: \"kubernetes.io/projected/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-api-access-sqgwt\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.746112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.746077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.847244 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.847217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.847364 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.847247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c211eb-deec-4c6b-a553-bbf965bbf480-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.847364 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.847298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45c211eb-deec-4c6b-a553-bbf965bbf480-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.847364 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.847321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgwt\" (UniqueName: \"kubernetes.io/projected/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-api-access-sqgwt\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.847364 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.847344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.847593 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:22:32.847570    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 17 17:22:32.847655 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:22:32.847646    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c211eb-deec-4c6b-a553-bbf965bbf480-proxy-tls podName:45c211eb-deec-4c6b-a553-bbf965bbf480 nodeName:}" failed. No retries permitted until 2026-09-17 17:22:33.347627628 +0000 UTC m=+2462.059634887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45c211eb-deec-4c6b-a553-bbf965bbf480-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" (UID: "45c211eb-deec-4c6b-a553-bbf965bbf480") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 17 17:22:32.847737 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.847717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45c211eb-deec-4c6b-a553-bbf965bbf480-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.847921 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.847906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.847964 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.847939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:32.855985 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:32.855968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgwt\" (UniqueName: \"kubernetes.io/projected/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-api-access-sqgwt\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:33.126732 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:33.126702    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:22:33.352169 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:33.352136    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c211eb-deec-4c6b-a553-bbf965bbf480-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:33.354556 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:33.354538    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c211eb-deec-4c6b-a553-bbf965bbf480-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:33.575020 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:33.574990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:33.854495 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:33.854470    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"]
Sep 17 17:22:33.856982 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:22:33.856957    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c211eb_deec_4c6b_a553_bbf965bbf480.slice/crio-cd02d12156111c393488c937365e8740e8d75a473b6b3f0bd846192afb6585e4 WatchSource:0}: Error finding container cd02d12156111c393488c937365e8740e8d75a473b6b3f0bd846192afb6585e4: Status 404 returned error can't find the container with id cd02d12156111c393488c937365e8740e8d75a473b6b3f0bd846192afb6585e4
Sep 17 17:22:34.028723 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:34.028672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" event={"ID":"45c211eb-deec-4c6b-a553-bbf965bbf480","Type":"ContainerStarted","Data":"42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b"}
Sep 17 17:22:34.028863 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:34.028755    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" event={"ID":"45c211eb-deec-4c6b-a553-bbf965bbf480","Type":"ContainerStarted","Data":"cd02d12156111c393488c937365e8740e8d75a473b6b3f0bd846192afb6585e4"}
Sep 17 17:22:39.056211 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:39.056182    2566 generic.go:356] "Generic (PLEG): container finished" podID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerID="42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b" exitCode=0
Sep 17 17:22:39.056611 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:39.056264    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" event={"ID":"45c211eb-deec-4c6b-a553-bbf965bbf480","Type":"ContainerDied","Data":"42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b"}
Sep 17 17:22:40.063264 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:40.063207    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" event={"ID":"45c211eb-deec-4c6b-a553-bbf965bbf480","Type":"ContainerStarted","Data":"29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745"}
Sep 17 17:22:40.063652 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:40.063320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" event={"ID":"45c211eb-deec-4c6b-a553-bbf965bbf480","Type":"ContainerStarted","Data":"2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db"}
Sep 17 17:22:40.063652 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:40.063578    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:40.063743 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:40.063712    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:40.084657 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:40.084609    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" podStartSLOduration=3.952988028 podStartE2EDuration="8.084595847s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.131607819s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:22:40.081909194 +0000 UTC m=+2468.793916463" watchObservedRunningTime="2026-09-17 17:22:40.084595847 +0000 UTC m=+2468.796603115"
Sep 17 17:22:45.925016 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:45.924965    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 17:22:46.073510 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:46.073482    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:22:55.914788 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:22:55.914752    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:23:02.958236 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:02.958199    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"]
Sep 17 17:23:02.958721 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:02.958492    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="kserve-container" containerID="cri-o://757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3" gracePeriod=30
Sep 17 17:23:02.958721 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:02.958492    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="kube-rbac-proxy" containerID="cri-o://10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec" gracePeriod=30
Sep 17 17:23:02.974538 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:02.974503    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:23:03.178492 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:03.178465    2566 generic.go:356] "Generic (PLEG): container finished" podID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerID="10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec" exitCode=2
Sep 17 17:23:03.178629 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:03.178521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" event={"ID":"75efa128-178a-42c3-853c-6537ab7e5dbc","Type":"ContainerDied","Data":"10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec"}
Sep 17 17:23:05.906752 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:05.906710    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.63:8643/healthz\": dial tcp 10.133.0.63:8643: connect: connection refused"
Sep 17 17:23:05.912203 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:05.912167    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.63:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 17 17:23:08.089550 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:08.089512    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:23:09.101783 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.101761    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:23:09.131484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.131463    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-cabundle-cert\") pod \"75efa128-178a-42c3-853c-6537ab7e5dbc\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") "
Sep 17 17:23:09.131608 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.131494    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75efa128-178a-42c3-853c-6537ab7e5dbc-kserve-provision-location\") pod \"75efa128-178a-42c3-853c-6537ab7e5dbc\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") "
Sep 17 17:23:09.131608 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.131535    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-rbac-proxy-sar-config\") pod \"75efa128-178a-42c3-853c-6537ab7e5dbc\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") "
Sep 17 17:23:09.131735 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.131628    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cndt4\" (UniqueName: \"kubernetes.io/projected/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-api-access-cndt4\") pod \"75efa128-178a-42c3-853c-6537ab7e5dbc\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") "
Sep 17 17:23:09.131735 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.131670    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75efa128-178a-42c3-853c-6537ab7e5dbc-proxy-tls\") pod \"75efa128-178a-42c3-853c-6537ab7e5dbc\" (UID: \"75efa128-178a-42c3-853c-6537ab7e5dbc\") "
Sep 17 17:23:09.131851 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.131813    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75efa128-178a-42c3-853c-6537ab7e5dbc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75efa128-178a-42c3-853c-6537ab7e5dbc" (UID: "75efa128-178a-42c3-853c-6537ab7e5dbc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:23:09.131906 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.131862    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75efa128-178a-42c3-853c-6537ab7e5dbc" (UID: "75efa128-178a-42c3-853c-6537ab7e5dbc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:23:09.131906 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.131867    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75efa128-178a-42c3-853c-6537ab7e5dbc" (UID: "75efa128-178a-42c3-853c-6537ab7e5dbc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:23:09.132078 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.132061    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:23:09.132132 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.132083    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75efa128-178a-42c3-853c-6537ab7e5dbc-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:23:09.132132 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.132100    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:23:09.133688 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.133666    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-api-access-cndt4" (OuterVolumeSpecName: "kube-api-access-cndt4") pod "75efa128-178a-42c3-853c-6537ab7e5dbc" (UID: "75efa128-178a-42c3-853c-6537ab7e5dbc"). InnerVolumeSpecName "kube-api-access-cndt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:23:09.133688 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.133674    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75efa128-178a-42c3-853c-6537ab7e5dbc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75efa128-178a-42c3-853c-6537ab7e5dbc" (UID: "75efa128-178a-42c3-853c-6537ab7e5dbc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:23:09.207626 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.207561    2566 generic.go:356] "Generic (PLEG): container finished" podID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerID="757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3" exitCode=0
Sep 17 17:23:09.207626 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.207608    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" event={"ID":"75efa128-178a-42c3-853c-6537ab7e5dbc","Type":"ContainerDied","Data":"757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3"}
Sep 17 17:23:09.207626 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.207613    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"
Sep 17 17:23:09.207809 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.207654    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p" event={"ID":"75efa128-178a-42c3-853c-6537ab7e5dbc","Type":"ContainerDied","Data":"e1a79dc350a810289c3bb06be95c919955927c86261b1880bd1ae46da634b4f0"}
Sep 17 17:23:09.207809 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.207679    2566 scope.go:117] "RemoveContainer" containerID="10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec"
Sep 17 17:23:09.216535 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.216516    2566 scope.go:117] "RemoveContainer" containerID="757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3"
Sep 17 17:23:09.224147 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.224127    2566 scope.go:117] "RemoveContainer" containerID="3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c"
Sep 17 17:23:09.229460 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.229432    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"]
Sep 17 17:23:09.231167 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.231145    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xtj5p"]
Sep 17 17:23:09.231958 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.231940    2566 scope.go:117] "RemoveContainer" containerID="10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec"
Sep 17 17:23:09.232223 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:23:09.232202    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec\": container with ID starting with 10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec not found: ID does not exist" containerID="10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec"
Sep 17 17:23:09.232301 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.232230    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec"} err="failed to get container status \"10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec\": rpc error: code = NotFound desc = could not find container \"10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec\": container with ID starting with 10e1f385aa884c29d6bff280f5dfab032f9e40dbe30b7075f39399c733915cec not found: ID does not exist"
Sep 17 17:23:09.232301 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.232244    2566 scope.go:117] "RemoveContainer" containerID="757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3"
Sep 17 17:23:09.232540 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:23:09.232519    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3\": container with ID starting with 757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3 not found: ID does not exist" containerID="757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3"
Sep 17 17:23:09.232590 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.232548    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3"} err="failed to get container status \"757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3\": rpc error: code = NotFound desc = could not find container \"757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3\": container with ID starting with 757badde52d6e0b1a93c492a657e4a4734ba0e0a3b414ad5dcf11b00cf2173d3 not found: ID does not exist"
Sep 17 17:23:09.232590 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.232569    2566 scope.go:117] "RemoveContainer" containerID="3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c"
Sep 17 17:23:09.232670 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.232652    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cndt4\" (UniqueName: \"kubernetes.io/projected/75efa128-178a-42c3-853c-6537ab7e5dbc-kube-api-access-cndt4\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:23:09.232731 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.232670    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75efa128-178a-42c3-853c-6537ab7e5dbc-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:23:09.232847 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:23:09.232832    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c\": container with ID starting with 3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c not found: ID does not exist" containerID="3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c"
Sep 17 17:23:09.232892 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.232851    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c"} err="failed to get container status \"3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c\": rpc error: code = NotFound desc = could not find container \"3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c\": container with ID starting with 3ae671ff658d965e1819e1912e68a638fe6db61439a7065d656dab2ed1e90d8c not found: ID does not exist"
Sep 17 17:23:09.378436 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.378410    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:23:09.809446 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:09.809409    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" path="/var/lib/kubelet/pods/75efa128-178a-42c3-853c-6537ab7e5dbc/volumes"
Sep 17 17:23:16.077523 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:16.077496    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:23:22.672802 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:22.672768    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"]
Sep 17 17:23:22.673729 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:22.673672    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" podUID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerName="kserve-container" containerID="cri-o://2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db" gracePeriod=30
Sep 17 17:23:22.674163 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:22.674108    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" podUID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerName="kube-rbac-proxy" containerID="cri-o://29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745" gracePeriod=30
Sep 17 17:23:22.688420 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:22.688393    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:23:23.278618 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.278585    2566 generic.go:356] "Generic (PLEG): container finished" podID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerID="29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745" exitCode=2
Sep 17 17:23:23.278805 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.278665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" event={"ID":"45c211eb-deec-4c6b-a553-bbf965bbf480","Type":"ContainerDied","Data":"29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745"}
Sep 17 17:23:23.730368 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.730347    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:23:23.845071 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.844999    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c211eb-deec-4c6b-a553-bbf965bbf480-proxy-tls\") pod \"45c211eb-deec-4c6b-a553-bbf965bbf480\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") "
Sep 17 17:23:23.845071 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.845048    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-rbac-proxy-sar-config\") pod \"45c211eb-deec-4c6b-a553-bbf965bbf480\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") "
Sep 17 17:23:23.845071 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.845073    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqgwt\" (UniqueName: \"kubernetes.io/projected/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-api-access-sqgwt\") pod \"45c211eb-deec-4c6b-a553-bbf965bbf480\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") "
Sep 17 17:23:23.845356 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.845106    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45c211eb-deec-4c6b-a553-bbf965bbf480-kserve-provision-location\") pod \"45c211eb-deec-4c6b-a553-bbf965bbf480\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") "
Sep 17 17:23:23.845356 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.845144    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-cabundle-cert\") pod \"45c211eb-deec-4c6b-a553-bbf965bbf480\" (UID: \"45c211eb-deec-4c6b-a553-bbf965bbf480\") "
Sep 17 17:23:23.845462 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.845365    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45c211eb-deec-4c6b-a553-bbf965bbf480" (UID: "45c211eb-deec-4c6b-a553-bbf965bbf480"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:23:23.845527 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.845494    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c211eb-deec-4c6b-a553-bbf965bbf480-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45c211eb-deec-4c6b-a553-bbf965bbf480" (UID: "45c211eb-deec-4c6b-a553-bbf965bbf480"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:23:23.845583 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.845544    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45c211eb-deec-4c6b-a553-bbf965bbf480" (UID: "45c211eb-deec-4c6b-a553-bbf965bbf480"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:23:23.847145 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.847128    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c211eb-deec-4c6b-a553-bbf965bbf480-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45c211eb-deec-4c6b-a553-bbf965bbf480" (UID: "45c211eb-deec-4c6b-a553-bbf965bbf480"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:23:23.847202 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.847126    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-api-access-sqgwt" (OuterVolumeSpecName: "kube-api-access-sqgwt") pod "45c211eb-deec-4c6b-a553-bbf965bbf480" (UID: "45c211eb-deec-4c6b-a553-bbf965bbf480"). InnerVolumeSpecName "kube-api-access-sqgwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:23:23.946087 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.946061    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45c211eb-deec-4c6b-a553-bbf965bbf480-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:23:23.946087 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.946086    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:23:23.946087 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.946096    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c211eb-deec-4c6b-a553-bbf965bbf480-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:23:23.946276 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.946104    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:23:23.946276 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:23.946114    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqgwt\" (UniqueName: \"kubernetes.io/projected/45c211eb-deec-4c6b-a553-bbf965bbf480-kube-api-access-sqgwt\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:23:24.285049 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.285013    2566 generic.go:356] "Generic (PLEG): container finished" podID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerID="2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db" exitCode=0
Sep 17 17:23:24.285236 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.285066    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"
Sep 17 17:23:24.285236 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.285067    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" event={"ID":"45c211eb-deec-4c6b-a553-bbf965bbf480","Type":"ContainerDied","Data":"2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db"}
Sep 17 17:23:24.285236 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.285146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl" event={"ID":"45c211eb-deec-4c6b-a553-bbf965bbf480","Type":"ContainerDied","Data":"cd02d12156111c393488c937365e8740e8d75a473b6b3f0bd846192afb6585e4"}
Sep 17 17:23:24.285236 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.285187    2566 scope.go:117] "RemoveContainer" containerID="29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745"
Sep 17 17:23:24.295012 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.294997    2566 scope.go:117] "RemoveContainer" containerID="2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db"
Sep 17 17:23:24.302850 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.302830    2566 scope.go:117] "RemoveContainer" containerID="42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b"
Sep 17 17:23:24.307684 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.307662    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"]
Sep 17 17:23:24.310918 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.310901    2566 scope.go:117] "RemoveContainer" containerID="29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745"
Sep 17 17:23:24.311194 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:23:24.311176    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745\": container with ID starting with 29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745 not found: ID does not exist" containerID="29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745"
Sep 17 17:23:24.311281 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.311203    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745"} err="failed to get container status \"29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745\": rpc error: code = NotFound desc = could not find container \"29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745\": container with ID starting with 29f945aa0a0b0c0fd4a5e04678844e4776985ff6766460c48388e98e454f9745 not found: ID does not exist"
Sep 17 17:23:24.311281 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.311225    2566 scope.go:117] "RemoveContainer" containerID="2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db"
Sep 17 17:23:24.311399 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.311383    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vjffl"]
Sep 17 17:23:24.311499 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:23:24.311481    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db\": container with ID starting with 2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db not found: ID does not exist" containerID="2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db"
Sep 17 17:23:24.311539 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.311506    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db"} err="failed to get container status \"2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db\": rpc error: code = NotFound desc = could not find container \"2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db\": container with ID starting with 2bbcab011a318d56a3a169edeae4379d615eaa58c460ff43c02b10f2656784db not found: ID does not exist"
Sep 17 17:23:24.311539 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.311523    2566 scope.go:117] "RemoveContainer" containerID="42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b"
Sep 17 17:23:24.311766 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:23:24.311749    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b\": container with ID starting with 42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b not found: ID does not exist" containerID="42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b"
Sep 17 17:23:24.311811 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:24.311772    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b"} err="failed to get container status \"42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b\": rpc error: code = NotFound desc = could not find container \"42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b\": container with ID starting with 42dfb8c73fb79194bb30346e217eb4120745f112c81d05ca9a1c97feba874b9b not found: ID does not exist"
Sep 17 17:23:25.810444 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:25.810409    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45c211eb-deec-4c6b-a553-bbf965bbf480" path="/var/lib/kubelet/pods/45c211eb-deec-4c6b-a553-bbf965bbf480/volumes"
Sep 17 17:23:27.849578 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.849547    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:23:27.885600 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.885570    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"]
Sep 17 17:23:27.885902 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.885889    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerName="storage-initializer"
Sep 17 17:23:27.885957 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.885906    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerName="storage-initializer"
Sep 17 17:23:27.885957 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.885925    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="kube-rbac-proxy"
Sep 17 17:23:27.885957 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.885934    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="kube-rbac-proxy"
Sep 17 17:23:27.885957 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.885955    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="storage-initializer"
Sep 17 17:23:27.886112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.885962    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="storage-initializer"
Sep 17 17:23:27.886112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.885974    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerName="kube-rbac-proxy"
Sep 17 17:23:27.886112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.885979    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerName="kube-rbac-proxy"
Sep 17 17:23:27.886112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.885989    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerName="kserve-container"
Sep 17 17:23:27.886112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.885994    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerName="kserve-container"
Sep 17 17:23:27.886112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.886001    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="kserve-container"
Sep 17 17:23:27.886112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.886006    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="kserve-container"
Sep 17 17:23:27.886112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.886081    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="kserve-container"
Sep 17 17:23:27.886112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.886091    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerName="kserve-container"
Sep 17 17:23:27.886112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.886098    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c211eb-deec-4c6b-a553-bbf965bbf480" containerName="kube-rbac-proxy"
Sep 17 17:23:27.886112 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.886104    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="75efa128-178a-42c3-853c-6537ab7e5dbc" containerName="kube-rbac-proxy"
Sep 17 17:23:27.890902 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.890883    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:27.893719 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.893693    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:23:27.893719 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.893715    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 17 17:23:27.894090 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.894075    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:23:27.895056 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.895037    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:23:27.895160 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.895067    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:23:27.895160 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.895109    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 17 17:23:27.895377 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.895363    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:23:27.900168 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.900149    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"]
Sep 17 17:23:27.976560 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.976528    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c945bdef-1d75-4727-b169-cf8723db655c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:27.976715 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.976590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c945bdef-1d75-4727-b169-cf8723db655c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:27.976715 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.976624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgfr6\" (UniqueName: \"kubernetes.io/projected/c945bdef-1d75-4727-b169-cf8723db655c-kube-api-access-kgfr6\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:27.976715 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.976650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:27.976715 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:27.976677    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.077443 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.077415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c945bdef-1d75-4727-b169-cf8723db655c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.077572 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.077461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c945bdef-1d75-4727-b169-cf8723db655c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.077572 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:23:28.077563    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 17 17:23:28.077653 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.077630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgfr6\" (UniqueName: \"kubernetes.io/projected/c945bdef-1d75-4727-b169-cf8723db655c-kube-api-access-kgfr6\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.077695 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.077681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.077739 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.077714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.077785 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:23:28.077742    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c945bdef-1d75-4727-b169-cf8723db655c-proxy-tls podName:c945bdef-1d75-4727-b169-cf8723db655c nodeName:}" failed. No retries permitted until 2026-09-17 17:23:28.577719034 +0000 UTC m=+2517.289726291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c945bdef-1d75-4727-b169-cf8723db655c-proxy-tls") pod "isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" (UID: "c945bdef-1d75-4727-b169-cf8723db655c") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 17 17:23:28.077847 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.077789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c945bdef-1d75-4727-b169-cf8723db655c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.078202 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.078186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.078245 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.078227    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.088324 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.088299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgfr6\" (UniqueName: \"kubernetes.io/projected/c945bdef-1d75-4727-b169-cf8723db655c-kube-api-access-kgfr6\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.357069 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.357029    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:23:28.582428 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.582397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c945bdef-1d75-4727-b169-cf8723db655c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.586066 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.586029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c945bdef-1d75-4727-b169-cf8723db655c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-75869c94b4-vwmwq\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:28.802571 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:28.802535    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:29.082419 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:29.082397    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"]
Sep 17 17:23:29.084142 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:23:29.084115    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc945bdef_1d75_4727_b169_cf8723db655c.slice/crio-1e6ce71c794683c452c4cc9b46da41223e98107d044a5c2f041d54e19b6e7ed4 WatchSource:0}: Error finding container 1e6ce71c794683c452c4cc9b46da41223e98107d044a5c2f041d54e19b6e7ed4: Status 404 returned error can't find the container with id 1e6ce71c794683c452c4cc9b46da41223e98107d044a5c2f041d54e19b6e7ed4
Sep 17 17:23:29.309586 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:29.309543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" event={"ID":"c945bdef-1d75-4727-b169-cf8723db655c","Type":"ContainerStarted","Data":"99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d"}
Sep 17 17:23:29.309745 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:29.309633    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" event={"ID":"c945bdef-1d75-4727-b169-cf8723db655c","Type":"ContainerStarted","Data":"1e6ce71c794683c452c4cc9b46da41223e98107d044a5c2f041d54e19b6e7ed4"}
Sep 17 17:23:33.332075 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:33.332041    2566 generic.go:356] "Generic (PLEG): container finished" podID="c945bdef-1d75-4727-b169-cf8723db655c" containerID="99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d" exitCode=0
Sep 17 17:23:33.332562 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:33.332119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" event={"ID":"c945bdef-1d75-4727-b169-cf8723db655c","Type":"ContainerDied","Data":"99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d"}
Sep 17 17:23:34.342095 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:34.342053    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" event={"ID":"c945bdef-1d75-4727-b169-cf8723db655c","Type":"ContainerStarted","Data":"45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365"}
Sep 17 17:23:36.354719 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:36.354691    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" event={"ID":"c945bdef-1d75-4727-b169-cf8723db655c","Type":"ContainerStarted","Data":"1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964"}
Sep 17 17:23:37.361948 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:37.361909    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" event={"ID":"c945bdef-1d75-4727-b169-cf8723db655c","Type":"ContainerStarted","Data":"6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3"}
Sep 17 17:23:37.362449 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:37.362175    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:37.362449 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:37.362231    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:37.362449 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:37.362272    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:37.384490 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:37.384444    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podStartSLOduration=3.740119091 podStartE2EDuration="10.384430478s" totalImagesPullingTime="2.88496788s" totalInitContainerRuntime="3.759343507s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:23:37.383099331 +0000 UTC m=+2526.095106598" watchObservedRunningTime="2026-09-17 17:23:37.384430478 +0000 UTC m=+2526.096437748"
Sep 17 17:23:43.371710 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:43.371680    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:23:43.453010 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:43.452976    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:23:43.503376 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:23:43.503343    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:24:03.373461 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:03.373423    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 17 17:24:13.374156 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:13.374124    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:24:28.746342 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:28.746302    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:24:33.922966 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:33.922934    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:24:34.167810 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:34.167776    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:24:43.375861 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:43.375830    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:24:47.888644 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:47.888614    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"]
Sep 17 17:24:47.889070 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:47.888826    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-container" containerID="cri-o://45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365" gracePeriod=30
Sep 17 17:24:47.889070 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:47.888889    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-agent" containerID="cri-o://1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964" gracePeriod=30
Sep 17 17:24:47.889070 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:47.888889    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kube-rbac-proxy" containerID="cri-o://6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3" gracePeriod=30
Sep 17 17:24:47.909000 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:47.908971    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:24:48.367446 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:48.367400    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Sep 17 17:24:48.716198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:48.716107    2566 generic.go:356] "Generic (PLEG): container finished" podID="c945bdef-1d75-4727-b169-cf8723db655c" containerID="6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3" exitCode=2
Sep 17 17:24:48.716198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:48.716174    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" event={"ID":"c945bdef-1d75-4727-b169-cf8723db655c","Type":"ContainerDied","Data":"6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3"}
Sep 17 17:24:50.728873 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:50.728834    2566 generic.go:356] "Generic (PLEG): container finished" podID="c945bdef-1d75-4727-b169-cf8723db655c" containerID="45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365" exitCode=0
Sep 17 17:24:50.729285 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:50.728905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" event={"ID":"c945bdef-1d75-4727-b169-cf8723db655c","Type":"ContainerDied","Data":"45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365"}
Sep 17 17:24:53.094521 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:53.094471    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:24:53.367429 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:53.367344    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Sep 17 17:24:53.372181 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:53.372153    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 17 17:24:53.399493 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:53.399469    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:24:57.744524 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:57.744432    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:24:58.366956 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:58.366920    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Sep 17 17:24:58.367130 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:24:58.367042    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:25:03.366840 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:03.366795    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Sep 17 17:25:03.372700 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:03.372667    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 17 17:25:08.367127 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:08.367075    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Sep 17 17:25:13.367262 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:13.367202    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Sep 17 17:25:13.372678 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:13.372649    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 17 17:25:13.372793 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:13.372778    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:25:18.069585 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.069561    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:25:18.120533 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.120507    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-kube-rbac-proxy-sar-config\") pod \"c945bdef-1d75-4727-b169-cf8723db655c\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") "
Sep 17 17:25:18.120679 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.120553    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgfr6\" (UniqueName: \"kubernetes.io/projected/c945bdef-1d75-4727-b169-cf8723db655c-kube-api-access-kgfr6\") pod \"c945bdef-1d75-4727-b169-cf8723db655c\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") "
Sep 17 17:25:18.120679 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.120586    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c945bdef-1d75-4727-b169-cf8723db655c-kserve-provision-location\") pod \"c945bdef-1d75-4727-b169-cf8723db655c\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") "
Sep 17 17:25:18.120679 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.120621    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c945bdef-1d75-4727-b169-cf8723db655c-proxy-tls\") pod \"c945bdef-1d75-4727-b169-cf8723db655c\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") "
Sep 17 17:25:18.120679 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.120668    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-cabundle-cert\") pod \"c945bdef-1d75-4727-b169-cf8723db655c\" (UID: \"c945bdef-1d75-4727-b169-cf8723db655c\") "
Sep 17 17:25:18.120941 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.120914    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c945bdef-1d75-4727-b169-cf8723db655c" (UID: "c945bdef-1d75-4727-b169-cf8723db655c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:25:18.120992 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.120937    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c945bdef-1d75-4727-b169-cf8723db655c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c945bdef-1d75-4727-b169-cf8723db655c" (UID: "c945bdef-1d75-4727-b169-cf8723db655c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:25:18.121042 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.121019    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c945bdef-1d75-4727-b169-cf8723db655c" (UID: "c945bdef-1d75-4727-b169-cf8723db655c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:25:18.122605 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.122583    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c945bdef-1d75-4727-b169-cf8723db655c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c945bdef-1d75-4727-b169-cf8723db655c" (UID: "c945bdef-1d75-4727-b169-cf8723db655c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:25:18.122792 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.122773    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c945bdef-1d75-4727-b169-cf8723db655c-kube-api-access-kgfr6" (OuterVolumeSpecName: "kube-api-access-kgfr6") pod "c945bdef-1d75-4727-b169-cf8723db655c" (UID: "c945bdef-1d75-4727-b169-cf8723db655c"). InnerVolumeSpecName "kube-api-access-kgfr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:25:18.221234 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.221184    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgfr6\" (UniqueName: \"kubernetes.io/projected/c945bdef-1d75-4727-b169-cf8723db655c-kube-api-access-kgfr6\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:25:18.221234 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.221205    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c945bdef-1d75-4727-b169-cf8723db655c-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:25:18.221234 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.221216    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c945bdef-1d75-4727-b169-cf8723db655c-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:25:18.221234 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.221225    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:25:18.221444 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.221233    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c945bdef-1d75-4727-b169-cf8723db655c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:25:18.866336 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.866299    2566 generic.go:356] "Generic (PLEG): container finished" podID="c945bdef-1d75-4727-b169-cf8723db655c" containerID="1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964" exitCode=137
Sep 17 17:25:18.866598 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.866366    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"
Sep 17 17:25:18.866598 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.866387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" event={"ID":"c945bdef-1d75-4727-b169-cf8723db655c","Type":"ContainerDied","Data":"1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964"}
Sep 17 17:25:18.866598 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.866455    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq" event={"ID":"c945bdef-1d75-4727-b169-cf8723db655c","Type":"ContainerDied","Data":"1e6ce71c794683c452c4cc9b46da41223e98107d044a5c2f041d54e19b6e7ed4"}
Sep 17 17:25:18.866598 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.866486    2566 scope.go:117] "RemoveContainer" containerID="6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3"
Sep 17 17:25:18.875348 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.875329    2566 scope.go:117] "RemoveContainer" containerID="1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964"
Sep 17 17:25:18.882729 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.882712    2566 scope.go:117] "RemoveContainer" containerID="45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365"
Sep 17 17:25:18.888629 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.888604    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"]
Sep 17 17:25:18.890641 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.890618    2566 scope.go:117] "RemoveContainer" containerID="99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d"
Sep 17 17:25:18.897807 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.897777    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-75869c94b4-vwmwq"]
Sep 17 17:25:18.902108 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.902047    2566 scope.go:117] "RemoveContainer" containerID="6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3"
Sep 17 17:25:18.902514 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:25:18.902474    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3\": container with ID starting with 6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3 not found: ID does not exist" containerID="6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3"
Sep 17 17:25:18.902576 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.902534    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3"} err="failed to get container status \"6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3\": rpc error: code = NotFound desc = could not find container \"6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3\": container with ID starting with 6db8e9abfc544598c5a2dbe91b917477a9d44e97c64d3365e135629bca84b3a3 not found: ID does not exist"
Sep 17 17:25:18.902576 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.902568    2566 scope.go:117] "RemoveContainer" containerID="1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964"
Sep 17 17:25:18.902899 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:25:18.902876    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964\": container with ID starting with 1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964 not found: ID does not exist" containerID="1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964"
Sep 17 17:25:18.902967 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.902924    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964"} err="failed to get container status \"1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964\": rpc error: code = NotFound desc = could not find container \"1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964\": container with ID starting with 1c1784b0839bedbc571944b185753aa7f0d9c7d5d2391168cdc34377794de964 not found: ID does not exist"
Sep 17 17:25:18.902967 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.902948    2566 scope.go:117] "RemoveContainer" containerID="45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365"
Sep 17 17:25:18.903218 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:25:18.903193    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365\": container with ID starting with 45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365 not found: ID does not exist" containerID="45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365"
Sep 17 17:25:18.903277 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.903222    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365"} err="failed to get container status \"45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365\": rpc error: code = NotFound desc = could not find container \"45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365\": container with ID starting with 45e6b6b3fe913b00ba1de84842cd5badfe4fbe5003ddb2f4c46035e4218a9365 not found: ID does not exist"
Sep 17 17:25:18.903277 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.903236    2566 scope.go:117] "RemoveContainer" containerID="99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d"
Sep 17 17:25:18.903547 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:25:18.903527    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d\": container with ID starting with 99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d not found: ID does not exist" containerID="99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d"
Sep 17 17:25:18.903610 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:18.903554    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d"} err="failed to get container status \"99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d\": rpc error: code = NotFound desc = could not find container \"99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d\": container with ID starting with 99676620b43d2ca347c5c4af105ffd3050f4d99b0b9dd73a7c560e10a0a1943d not found: ID does not exist"
Sep 17 17:25:19.810877 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:19.810829    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c945bdef-1d75-4727-b169-cf8723db655c" path="/var/lib/kubelet/pods/c945bdef-1d75-4727-b169-cf8723db655c/volumes"
Sep 17 17:25:54.045301 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.045212    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:25:54.169543 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.169483    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:25:54.198517 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.198486    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"]
Sep 17 17:25:54.198852 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.198840    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="storage-initializer"
Sep 17 17:25:54.198905 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.198860    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="storage-initializer"
Sep 17 17:25:54.198905 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.198869    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-agent"
Sep 17 17:25:54.198905 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.198874    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-agent"
Sep 17 17:25:54.198905 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.198881    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-container"
Sep 17 17:25:54.198905 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.198886    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-container"
Sep 17 17:25:54.199075 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.198922    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kube-rbac-proxy"
Sep 17 17:25:54.199075 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.198938    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kube-rbac-proxy"
Sep 17 17:25:54.199075 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.199008    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-agent"
Sep 17 17:25:54.199075 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.199017    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kserve-container"
Sep 17 17:25:54.199075 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.199024    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c945bdef-1d75-4727-b169-cf8723db655c" containerName="kube-rbac-proxy"
Sep 17 17:25:54.202408 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.202390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.204984 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.204962    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:25:54.205401 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.204969    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 17 17:25:54.205401 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.205138    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 17:25:54.205401 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.205287    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:25:54.205620 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.205523    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:25:54.206037 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.206018    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 17 17:25:54.206132 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.206042    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:25:54.211197 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.211177    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"]
Sep 17 17:25:54.280681 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.280651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.280832 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.280706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8nn6\" (UniqueName: \"kubernetes.io/projected/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-api-access-x8nn6\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.280832 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.280771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f226e1fe-f964-4ba8-a60e-21fb25d56e51-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.280912 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.280860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.280912 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.280898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.303587 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.303514    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:25:54.381509 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.381483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.381624 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.381513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.381670 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.381649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.381714 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.381702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8nn6\" (UniqueName: \"kubernetes.io/projected/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-api-access-x8nn6\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.381816 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.381746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f226e1fe-f964-4ba8-a60e-21fb25d56e51-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.381916 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.381897    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.382218 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.382200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.382324 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.382302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.384196 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.384178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f226e1fe-f964-4ba8-a60e-21fb25d56e51-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.389800 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.389777    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8nn6\" (UniqueName: \"kubernetes.io/projected/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-api-access-x8nn6\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.514632 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.514606    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:54.790516 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:54.790491    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"]
Sep 17 17:25:54.792693 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:25:54.792665    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf226e1fe_f964_4ba8_a60e_21fb25d56e51.slice/crio-69489249b65ae836ec806da832e73d4a8f60a7aa73cc9f41dfb57de5ef2fa4e0 WatchSource:0}: Error finding container 69489249b65ae836ec806da832e73d4a8f60a7aa73cc9f41dfb57de5ef2fa4e0: Status 404 returned error can't find the container with id 69489249b65ae836ec806da832e73d4a8f60a7aa73cc9f41dfb57de5ef2fa4e0
Sep 17 17:25:55.035154 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:55.035118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" event={"ID":"f226e1fe-f964-4ba8-a60e-21fb25d56e51","Type":"ContainerStarted","Data":"34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507"}
Sep 17 17:25:55.035358 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:55.035173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" event={"ID":"f226e1fe-f964-4ba8-a60e-21fb25d56e51","Type":"ContainerStarted","Data":"69489249b65ae836ec806da832e73d4a8f60a7aa73cc9f41dfb57de5ef2fa4e0"}
Sep 17 17:25:56.041266 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:56.041225    2566 generic.go:356] "Generic (PLEG): container finished" podID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerID="34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507" exitCode=0
Sep 17 17:25:56.041582 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:56.041287    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" event={"ID":"f226e1fe-f964-4ba8-a60e-21fb25d56e51","Type":"ContainerDied","Data":"34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507"}
Sep 17 17:25:57.048561 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:57.048525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" event={"ID":"f226e1fe-f964-4ba8-a60e-21fb25d56e51","Type":"ContainerStarted","Data":"e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f"}
Sep 17 17:25:57.048959 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:57.048592    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" event={"ID":"f226e1fe-f964-4ba8-a60e-21fb25d56e51","Type":"ContainerStarted","Data":"2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b"}
Sep 17 17:25:57.048959 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:57.048704    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:57.048959 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:57.048728    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:25:57.049977 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:57.049951    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 17 17:25:57.070719 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:57.070667    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podStartSLOduration=1.907325599 podStartE2EDuration="3.070652217s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.163326618s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:25:57.068982544 +0000 UTC m=+2665.780989813" watchObservedRunningTime="2026-09-17 17:25:57.070652217 +0000 UTC m=+2665.782659484"
Sep 17 17:25:58.060557 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:25:58.060515    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 17 17:26:03.065454 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:26:03.065427    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:26:03.065913 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:26:03.065889    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 17 17:26:07.235129 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:26:07.235072    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:26:08.562580 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:26:08.562544    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:26:13.066769 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:26:13.066729    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 17 17:26:13.694459 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:26:13.694422    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:26:14.397931 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:26:14.397891    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:26:23.066078 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:26:23.066039    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 17 17:26:33.066411 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:26:33.066324    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 17 17:26:43.066573 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:26:43.066530    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 17 17:26:53.065859 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:26:53.065822    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 17 17:27:03.066754 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:03.066709    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 17 17:27:13.066488 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:13.066444    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:27:14.213376 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:14.213341    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"]
Sep 17 17:27:14.213756 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:14.213576    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container" containerID="cri-o://2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b" gracePeriod=30
Sep 17 17:27:14.213756 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:14.213614    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kube-rbac-proxy" containerID="cri-o://e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f" gracePeriod=30
Sep 17 17:27:14.225757 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:14.225730    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:27:14.430113 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:14.430085    2566 generic.go:356] "Generic (PLEG): container finished" podID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerID="e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f" exitCode=2
Sep 17 17:27:14.430238 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:14.430134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" event={"ID":"f226e1fe-f964-4ba8-a60e-21fb25d56e51","Type":"ContainerDied","Data":"e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f"}
Sep 17 17:27:15.255477 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:15.255432    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:27:15.357552 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:15.357520    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:27:15.761268 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:15.761222    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:27:18.061074 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.061035    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Sep 17 17:27:18.351863 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.351840    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:27:18.418990 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.418965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8nn6\" (UniqueName: \"kubernetes.io/projected/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-api-access-x8nn6\") pod \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") "
Sep 17 17:27:18.418990 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.418994    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f226e1fe-f964-4ba8-a60e-21fb25d56e51-proxy-tls\") pod \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") "
Sep 17 17:27:18.419332 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.419014    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-cabundle-cert\") pod \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") "
Sep 17 17:27:18.419332 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.419043    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kserve-provision-location\") pod \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") "
Sep 17 17:27:18.419332 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.419088    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-rbac-proxy-sar-config\") pod \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\" (UID: \"f226e1fe-f964-4ba8-a60e-21fb25d56e51\") "
Sep 17 17:27:18.419492 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.419412    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f226e1fe-f964-4ba8-a60e-21fb25d56e51" (UID: "f226e1fe-f964-4ba8-a60e-21fb25d56e51"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:27:18.419492 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.419421    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f226e1fe-f964-4ba8-a60e-21fb25d56e51" (UID: "f226e1fe-f964-4ba8-a60e-21fb25d56e51"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:27:18.419492 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.419456    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f226e1fe-f964-4ba8-a60e-21fb25d56e51" (UID: "f226e1fe-f964-4ba8-a60e-21fb25d56e51"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:27:18.421108 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.421088    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-api-access-x8nn6" (OuterVolumeSpecName: "kube-api-access-x8nn6") pod "f226e1fe-f964-4ba8-a60e-21fb25d56e51" (UID: "f226e1fe-f964-4ba8-a60e-21fb25d56e51"). InnerVolumeSpecName "kube-api-access-x8nn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:27:18.421108 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.421095    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f226e1fe-f964-4ba8-a60e-21fb25d56e51-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f226e1fe-f964-4ba8-a60e-21fb25d56e51" (UID: "f226e1fe-f964-4ba8-a60e-21fb25d56e51"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:27:18.451220 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.451190    2566 generic.go:356] "Generic (PLEG): container finished" podID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerID="2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b" exitCode=0
Sep 17 17:27:18.451342 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.451272    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"
Sep 17 17:27:18.451406 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.451272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" event={"ID":"f226e1fe-f964-4ba8-a60e-21fb25d56e51","Type":"ContainerDied","Data":"2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b"}
Sep 17 17:27:18.451469 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.451425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2" event={"ID":"f226e1fe-f964-4ba8-a60e-21fb25d56e51","Type":"ContainerDied","Data":"69489249b65ae836ec806da832e73d4a8f60a7aa73cc9f41dfb57de5ef2fa4e0"}
Sep 17 17:27:18.451522 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.451477    2566 scope.go:117] "RemoveContainer" containerID="e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f"
Sep 17 17:27:18.460366 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.460349    2566 scope.go:117] "RemoveContainer" containerID="2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b"
Sep 17 17:27:18.467995 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.467978    2566 scope.go:117] "RemoveContainer" containerID="34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507"
Sep 17 17:27:18.474644 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.474621    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"]
Sep 17 17:27:18.475544 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.475531    2566 scope.go:117] "RemoveContainer" containerID="e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f"
Sep 17 17:27:18.475791 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:27:18.475773    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f\": container with ID starting with e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f not found: ID does not exist" containerID="e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f"
Sep 17 17:27:18.475856 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.475797    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f"} err="failed to get container status \"e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f\": rpc error: code = NotFound desc = could not find container \"e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f\": container with ID starting with e03adbe4f40773bd53e00c5768865428effc4e9bd0fb740edaa03ca6357d209f not found: ID does not exist"
Sep 17 17:27:18.475856 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.475812    2566 scope.go:117] "RemoveContainer" containerID="2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b"
Sep 17 17:27:18.476079 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:27:18.476063    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b\": container with ID starting with 2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b not found: ID does not exist" containerID="2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b"
Sep 17 17:27:18.476132 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.476085    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b"} err="failed to get container status \"2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b\": rpc error: code = NotFound desc = could not find container \"2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b\": container with ID starting with 2781794a14d54ae641ccbc412eddfc04e24c4721fc12a2042958b873d5dcc73b not found: ID does not exist"
Sep 17 17:27:18.476132 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.476097    2566 scope.go:117] "RemoveContainer" containerID="34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507"
Sep 17 17:27:18.476350 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:27:18.476332    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507\": container with ID starting with 34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507 not found: ID does not exist" containerID="34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507"
Sep 17 17:27:18.476433 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.476359    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507"} err="failed to get container status \"34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507\": rpc error: code = NotFound desc = could not find container \"34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507\": container with ID starting with 34d13c7cff3e324cc0e5744c9c4983885290510044f35cf11b5d4f193b986507 not found: ID does not exist"
Sep 17 17:27:18.480217 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.480196    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64d978b4bf-54ks2"]
Sep 17 17:27:18.520355 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.520338    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:27:18.520435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.520358    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8nn6\" (UniqueName: \"kubernetes.io/projected/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kube-api-access-x8nn6\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:27:18.520435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.520369    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f226e1fe-f964-4ba8-a60e-21fb25d56e51-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:27:18.520435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.520378    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f226e1fe-f964-4ba8-a60e-21fb25d56e51-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:27:18.520435 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:18.520386    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f226e1fe-f964-4ba8-a60e-21fb25d56e51-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:27:19.809246 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:19.809209    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" path="/var/lib/kubelet/pods/f226e1fe-f964-4ba8-a60e-21fb25d56e51/volumes"
Sep 17 17:27:20.379515 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:20.379474    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:27:20.479806 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:20.479774    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:27:25.282899 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:25.282865    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:27:26.407676 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:26.407639    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:27:26.550836 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:27:26.550804    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:28:22.067236 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:22.067205    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:28:27.181526 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:27.181480    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:28:27.292047 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:27.292007    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:28:56.453588 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:56.453549    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:28:57.551668 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.551630    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:28:57.561892 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.561861    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"]
Sep 17 17:28:57.562223 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.562210    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container"
Sep 17 17:28:57.562292 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.562225    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container"
Sep 17 17:28:57.562292 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.562235    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kube-rbac-proxy"
Sep 17 17:28:57.562292 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.562241    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kube-rbac-proxy"
Sep 17 17:28:57.562292 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.562269    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="storage-initializer"
Sep 17 17:28:57.562292 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.562279    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="storage-initializer"
Sep 17 17:28:57.562464 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.562346    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kube-rbac-proxy"
Sep 17 17:28:57.562464 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.562360    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f226e1fe-f964-4ba8-a60e-21fb25d56e51" containerName="kserve-container"
Sep 17 17:28:57.565611 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.565597    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.568060 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.568038    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 17:28:57.568339 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.568299    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:28:57.568339 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.568333    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 17:28:57.568507 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.568341    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:28:57.568507 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.568355    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 17:28:57.568507 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.568302    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:28:57.575711 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.575692    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"]
Sep 17 17:28:57.652445 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.652417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lt2l\" (UniqueName: \"kubernetes.io/projected/141ac433-5edf-4351-a222-dde39594bedf-kube-api-access-2lt2l\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.652549 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.652462    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141ac433-5edf-4351-a222-dde39594bedf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.652549 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.652487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141ac433-5edf-4351-a222-dde39594bedf-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.652549 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.652522    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/141ac433-5edf-4351-a222-dde39594bedf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.753826 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.753798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141ac433-5edf-4351-a222-dde39594bedf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.753942 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.753835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141ac433-5edf-4351-a222-dde39594bedf-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.753942 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.753870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/141ac433-5edf-4351-a222-dde39594bedf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.754060 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.753945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lt2l\" (UniqueName: \"kubernetes.io/projected/141ac433-5edf-4351-a222-dde39594bedf-kube-api-access-2lt2l\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.754060 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:28:57.753971    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 17:28:57.754060 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:28:57.754058    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141ac433-5edf-4351-a222-dde39594bedf-proxy-tls podName:141ac433-5edf-4351-a222-dde39594bedf nodeName:}" failed. No retries permitted until 2026-09-17 17:28:58.254026542 +0000 UTC m=+2846.966033802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/141ac433-5edf-4351-a222-dde39594bedf-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb" (UID: "141ac433-5edf-4351-a222-dde39594bedf") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 17:28:57.754326 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.754305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/141ac433-5edf-4351-a222-dde39594bedf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.754484 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.754464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141ac433-5edf-4351-a222-dde39594bedf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.762428 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.762408    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lt2l\" (UniqueName: \"kubernetes.io/projected/141ac433-5edf-4351-a222-dde39594bedf-kube-api-access-2lt2l\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:57.821127 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:57.821059    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:28:58.258041 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:58.258012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141ac433-5edf-4351-a222-dde39594bedf-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:58.260336 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:58.260312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141ac433-5edf-4351-a222-dde39594bedf-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:58.478228 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:58.478192    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:28:58.752177 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:58.752153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"]
Sep 17 17:28:58.753788 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:28:58.753758    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141ac433_5edf_4351_a222_dde39594bedf.slice/crio-acc4f72a4fa66f1e6fd4c79b4813434a61450e39fa5591ef343e89df4d875e77 WatchSource:0}: Error finding container acc4f72a4fa66f1e6fd4c79b4813434a61450e39fa5591ef343e89df4d875e77: Status 404 returned error can't find the container with id acc4f72a4fa66f1e6fd4c79b4813434a61450e39fa5591ef343e89df4d875e77
Sep 17 17:28:58.917586 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:58.917555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb" event={"ID":"141ac433-5edf-4351-a222-dde39594bedf","Type":"ContainerStarted","Data":"ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49"}
Sep 17 17:28:58.917760 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:28:58.917614    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb" event={"ID":"141ac433-5edf-4351-a222-dde39594bedf","Type":"ContainerStarted","Data":"acc4f72a4fa66f1e6fd4c79b4813434a61450e39fa5591ef343e89df4d875e77"}
Sep 17 17:29:05.950981 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:05.950950    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb_141ac433-5edf-4351-a222-dde39594bedf/storage-initializer/0.log"
Sep 17 17:29:05.951356 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:05.950988    2566 generic.go:356] "Generic (PLEG): container finished" podID="141ac433-5edf-4351-a222-dde39594bedf" containerID="ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49" exitCode=1
Sep 17 17:29:05.951356 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:05.951061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb" event={"ID":"141ac433-5edf-4351-a222-dde39594bedf","Type":"ContainerDied","Data":"ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49"}
Sep 17 17:29:06.957408 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:06.957382    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb_141ac433-5edf-4351-a222-dde39594bedf/storage-initializer/0.log"
Sep 17 17:29:06.957779 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:06.957439    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb" event={"ID":"141ac433-5edf-4351-a222-dde39594bedf","Type":"ContainerStarted","Data":"9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a"}
Sep 17 17:29:07.497978 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:07.497948    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"]
Sep 17 17:29:07.519174 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:07.519141    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:29:07.962190 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:07.962093    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb" podUID="141ac433-5edf-4351-a222-dde39594bedf" containerName="storage-initializer" containerID="cri-o://9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a" gracePeriod=30
Sep 17 17:29:08.605668 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:08.605631    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:29:08.895801 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:08.895721    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:29:12.721605 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.721580    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb_141ac433-5edf-4351-a222-dde39594bedf/storage-initializer/1.log"
Sep 17 17:29:12.721970 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.721956    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb_141ac433-5edf-4351-a222-dde39594bedf/storage-initializer/0.log"
Sep 17 17:29:12.724100 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.724084    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:29:12.777051 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.777027    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/141ac433-5edf-4351-a222-dde39594bedf-kserve-provision-location\") pod \"141ac433-5edf-4351-a222-dde39594bedf\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") "
Sep 17 17:29:12.777161 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.777094    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141ac433-5edf-4351-a222-dde39594bedf-kube-rbac-proxy-sar-config\") pod \"141ac433-5edf-4351-a222-dde39594bedf\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") "
Sep 17 17:29:12.777161 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.777121    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lt2l\" (UniqueName: \"kubernetes.io/projected/141ac433-5edf-4351-a222-dde39594bedf-kube-api-access-2lt2l\") pod \"141ac433-5edf-4351-a222-dde39594bedf\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") "
Sep 17 17:29:12.777161 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.777151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141ac433-5edf-4351-a222-dde39594bedf-proxy-tls\") pod \"141ac433-5edf-4351-a222-dde39594bedf\" (UID: \"141ac433-5edf-4351-a222-dde39594bedf\") "
Sep 17 17:29:12.777361 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.777280    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141ac433-5edf-4351-a222-dde39594bedf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "141ac433-5edf-4351-a222-dde39594bedf" (UID: "141ac433-5edf-4351-a222-dde39594bedf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:12.777456 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.777437    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/141ac433-5edf-4351-a222-dde39594bedf-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:29:12.777539 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.777517    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141ac433-5edf-4351-a222-dde39594bedf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "141ac433-5edf-4351-a222-dde39594bedf" (UID: "141ac433-5edf-4351-a222-dde39594bedf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:29:12.779113 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.779095    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141ac433-5edf-4351-a222-dde39594bedf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "141ac433-5edf-4351-a222-dde39594bedf" (UID: "141ac433-5edf-4351-a222-dde39594bedf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:29:12.779231 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.779212    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141ac433-5edf-4351-a222-dde39594bedf-kube-api-access-2lt2l" (OuterVolumeSpecName: "kube-api-access-2lt2l") pod "141ac433-5edf-4351-a222-dde39594bedf" (UID: "141ac433-5edf-4351-a222-dde39594bedf"). InnerVolumeSpecName "kube-api-access-2lt2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:29:12.878181 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.878002    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/141ac433-5edf-4351-a222-dde39594bedf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:29:12.878181 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.878032    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lt2l\" (UniqueName: \"kubernetes.io/projected/141ac433-5edf-4351-a222-dde39594bedf-kube-api-access-2lt2l\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:29:12.878181 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.878047    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/141ac433-5edf-4351-a222-dde39594bedf-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:29:12.985907 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.985883    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb_141ac433-5edf-4351-a222-dde39594bedf/storage-initializer/1.log"
Sep 17 17:29:12.986242 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.986226    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb_141ac433-5edf-4351-a222-dde39594bedf/storage-initializer/0.log"
Sep 17 17:29:12.986349 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.986290    2566 generic.go:356] "Generic (PLEG): container finished" podID="141ac433-5edf-4351-a222-dde39594bedf" containerID="9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a" exitCode=1
Sep 17 17:29:12.986461 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.986341    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"
Sep 17 17:29:12.986461 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.986378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb" event={"ID":"141ac433-5edf-4351-a222-dde39594bedf","Type":"ContainerDied","Data":"9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a"}
Sep 17 17:29:12.986461 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.986437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb" event={"ID":"141ac433-5edf-4351-a222-dde39594bedf","Type":"ContainerDied","Data":"acc4f72a4fa66f1e6fd4c79b4813434a61450e39fa5591ef343e89df4d875e77"}
Sep 17 17:29:12.986589 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.986465    2566 scope.go:117] "RemoveContainer" containerID="9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a"
Sep 17 17:29:12.995586 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:12.995571    2566 scope.go:117] "RemoveContainer" containerID="ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49"
Sep 17 17:29:13.003026 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:13.003010    2566 scope.go:117] "RemoveContainer" containerID="9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a"
Sep 17 17:29:13.003296 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:29:13.003276    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a\": container with ID starting with 9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a not found: ID does not exist" containerID="9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a"
Sep 17 17:29:13.003376 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:13.003301    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a"} err="failed to get container status \"9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a\": rpc error: code = NotFound desc = could not find container \"9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a\": container with ID starting with 9eaebd46d35e7e59fc029c05764f1f9b51b5647ba306bc2f1544595fccac276a not found: ID does not exist"
Sep 17 17:29:13.003376 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:13.003316    2566 scope.go:117] "RemoveContainer" containerID="ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49"
Sep 17 17:29:13.003553 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:29:13.003536    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49\": container with ID starting with ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49 not found: ID does not exist" containerID="ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49"
Sep 17 17:29:13.003601 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:13.003558    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49"} err="failed to get container status \"ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49\": rpc error: code = NotFound desc = could not find container \"ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49\": container with ID starting with ed4308cc68cadc3032b550325d21e0db29ce6271848803b0f5a62780fa7c9b49 not found: ID does not exist"
Sep 17 17:29:13.029199 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:13.029176    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"]
Sep 17 17:29:13.032889 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:13.032866    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b8546b897-jvmgb"]
Sep 17 17:29:13.810310 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:29:13.810278    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141ac433-5edf-4351-a222-dde39594bedf" path="/var/lib/kubelet/pods/141ac433-5edf-4351-a222-dde39594bedf/volumes"
Sep 17 17:30:08.749506 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:08.749469    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:30:13.886017 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:13.885975    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:30:13.964665 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:13.964625    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:30:38.645915 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:38.645875    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:30:39.718362 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.718328    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:30:39.739598 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.739569    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"]
Sep 17 17:30:39.739936 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.739924    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141ac433-5edf-4351-a222-dde39594bedf" containerName="storage-initializer"
Sep 17 17:30:39.739987 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.739938    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="141ac433-5edf-4351-a222-dde39594bedf" containerName="storage-initializer"
Sep 17 17:30:39.739987 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.739949    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141ac433-5edf-4351-a222-dde39594bedf" containerName="storage-initializer"
Sep 17 17:30:39.739987 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.739955    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="141ac433-5edf-4351-a222-dde39594bedf" containerName="storage-initializer"
Sep 17 17:30:39.740114 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.740011    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="141ac433-5edf-4351-a222-dde39594bedf" containerName="storage-initializer"
Sep 17 17:30:39.740153 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.740114    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="141ac433-5edf-4351-a222-dde39594bedf" containerName="storage-initializer"
Sep 17 17:30:39.743307 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.743286    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:39.745914 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.745879    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 17:30:39.746024 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.745929    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 17:30:39.746024 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.745976    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:30:39.746024 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.746017    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:30:39.746157 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.746126    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:30:39.746839 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.746824    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 17:30:39.751471 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.751451    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"]
Sep 17 17:30:39.841640 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.841614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:39.841744 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.841678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:39.841795 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.841754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:39.841795 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.841777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqfs7\" (UniqueName: \"kubernetes.io/projected/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-api-access-tqfs7\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:39.942905 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.942867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:39.943030 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.942929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:39.943030 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.942958    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqfs7\" (UniqueName: \"kubernetes.io/projected/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-api-access-tqfs7\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:39.943030 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.943005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:39.943030 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:30:39.943018    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 17:30:39.943225 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:30:39.943096    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-proxy-tls podName:1f21c7a1-40f4-4151-aab6-913b6c5f54d1 nodeName:}" failed. No retries permitted until 2026-09-17 17:30:40.443076456 +0000 UTC m=+2949.155083727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns" (UID: "1f21c7a1-40f4-4151-aab6-913b6c5f54d1") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 17:30:39.943465 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.943446    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:39.943621 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.943604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:39.951432 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:39.951407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqfs7\" (UniqueName: \"kubernetes.io/projected/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-api-access-tqfs7\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:40.188317 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:40.188283    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:30:40.447957 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:40.447890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:40.450323 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:40.450295    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:40.654730 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:40.654697    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:40.927938 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:40.927912    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"]
Sep 17 17:30:40.930100 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:30:40.930069    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f21c7a1_40f4_4151_aab6_913b6c5f54d1.slice/crio-41575eb875e84825d6fe73ce38735453716dba8dfbd767c0fde2224fdf97d6a3 WatchSource:0}: Error finding container 41575eb875e84825d6fe73ce38735453716dba8dfbd767c0fde2224fdf97d6a3: Status 404 returned error can't find the container with id 41575eb875e84825d6fe73ce38735453716dba8dfbd767c0fde2224fdf97d6a3
Sep 17 17:30:41.397873 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:41.397838    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns" event={"ID":"1f21c7a1-40f4-4151-aab6-913b6c5f54d1","Type":"ContainerStarted","Data":"095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50"}
Sep 17 17:30:41.398035 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:41.397897    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns" event={"ID":"1f21c7a1-40f4-4151-aab6-913b6c5f54d1","Type":"ContainerStarted","Data":"41575eb875e84825d6fe73ce38735453716dba8dfbd767c0fde2224fdf97d6a3"}
Sep 17 17:30:46.423757 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:46.423732    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns_1f21c7a1-40f4-4151-aab6-913b6c5f54d1/storage-initializer/0.log"
Sep 17 17:30:46.424070 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:46.423772    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f21c7a1-40f4-4151-aab6-913b6c5f54d1" containerID="095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50" exitCode=1
Sep 17 17:30:46.424070 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:46.423853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns" event={"ID":"1f21c7a1-40f4-4151-aab6-913b6c5f54d1","Type":"ContainerDied","Data":"095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50"}
Sep 17 17:30:47.432368 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:47.432341    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns_1f21c7a1-40f4-4151-aab6-913b6c5f54d1/storage-initializer/0.log"
Sep 17 17:30:47.432752 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:47.432416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns" event={"ID":"1f21c7a1-40f4-4151-aab6-913b6c5f54d1","Type":"ContainerStarted","Data":"c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef"}
Sep 17 17:30:49.716627 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:49.716595    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"]
Sep 17 17:30:49.717077 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:49.716861    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns" podUID="1f21c7a1-40f4-4151-aab6-913b6c5f54d1" containerName="storage-initializer" containerID="cri-o://c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef" gracePeriod=30
Sep 17 17:30:49.733057 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:49.733029    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:30:50.359619 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.359600    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns_1f21c7a1-40f4-4151-aab6-913b6c5f54d1/storage-initializer/1.log"
Sep 17 17:30:50.360020 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.360002    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns_1f21c7a1-40f4-4151-aab6-913b6c5f54d1/storage-initializer/0.log"
Sep 17 17:30:50.362224 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.362208    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:50.428086 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.428062    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-rbac-proxy-sar-config\") pod \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") "
Sep 17 17:30:50.428186 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.428101    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kserve-provision-location\") pod \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") "
Sep 17 17:30:50.428186 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.428121    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqfs7\" (UniqueName: \"kubernetes.io/projected/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-api-access-tqfs7\") pod \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") "
Sep 17 17:30:50.428309 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.428230    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-proxy-tls\") pod \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\" (UID: \"1f21c7a1-40f4-4151-aab6-913b6c5f54d1\") "
Sep 17 17:30:50.428385 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.428363    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f21c7a1-40f4-4151-aab6-913b6c5f54d1" (UID: "1f21c7a1-40f4-4151-aab6-913b6c5f54d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:30:50.428488 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.428464    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f21c7a1-40f4-4151-aab6-913b6c5f54d1" (UID: "1f21c7a1-40f4-4151-aab6-913b6c5f54d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:30:50.428549 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.428536    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:30:50.428591 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.428555    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:30:50.430099 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.430079    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f21c7a1-40f4-4151-aab6-913b6c5f54d1" (UID: "1f21c7a1-40f4-4151-aab6-913b6c5f54d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:30:50.430159 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.430099    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-api-access-tqfs7" (OuterVolumeSpecName: "kube-api-access-tqfs7") pod "1f21c7a1-40f4-4151-aab6-913b6c5f54d1" (UID: "1f21c7a1-40f4-4151-aab6-913b6c5f54d1"). InnerVolumeSpecName "kube-api-access-tqfs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:30:50.446907 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.446886    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns_1f21c7a1-40f4-4151-aab6-913b6c5f54d1/storage-initializer/1.log"
Sep 17 17:30:50.447243 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.447230    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns_1f21c7a1-40f4-4151-aab6-913b6c5f54d1/storage-initializer/0.log"
Sep 17 17:30:50.447315 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.447284    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f21c7a1-40f4-4151-aab6-913b6c5f54d1" containerID="c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef" exitCode=1
Sep 17 17:30:50.447357 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.447343    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"
Sep 17 17:30:50.447392 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.447369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns" event={"ID":"1f21c7a1-40f4-4151-aab6-913b6c5f54d1","Type":"ContainerDied","Data":"c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef"}
Sep 17 17:30:50.447457 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.447446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns" event={"ID":"1f21c7a1-40f4-4151-aab6-913b6c5f54d1","Type":"ContainerDied","Data":"41575eb875e84825d6fe73ce38735453716dba8dfbd767c0fde2224fdf97d6a3"}
Sep 17 17:30:50.447493 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.447476    2566 scope.go:117] "RemoveContainer" containerID="c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef"
Sep 17 17:30:50.456146 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.456131    2566 scope.go:117] "RemoveContainer" containerID="095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50"
Sep 17 17:30:50.463226 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.463209    2566 scope.go:117] "RemoveContainer" containerID="c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef"
Sep 17 17:30:50.463481 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:30:50.463464    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef\": container with ID starting with c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef not found: ID does not exist" containerID="c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef"
Sep 17 17:30:50.463535 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.463487    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef"} err="failed to get container status \"c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef\": rpc error: code = NotFound desc = could not find container \"c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef\": container with ID starting with c5c8bb28fbe69859b7202e6530ad6106c6e13d2cd646b54f695c6488dcf53cef not found: ID does not exist"
Sep 17 17:30:50.463535 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.463500    2566 scope.go:117] "RemoveContainer" containerID="095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50"
Sep 17 17:30:50.463705 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:30:50.463688    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50\": container with ID starting with 095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50 not found: ID does not exist" containerID="095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50"
Sep 17 17:30:50.463742 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.463712    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50"} err="failed to get container status \"095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50\": rpc error: code = NotFound desc = could not find container \"095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50\": container with ID starting with 095db728196deb838b47cd931ff8ad2ad762892c862967ab2ef83945c4735e50 not found: ID does not exist"
Sep 17 17:30:50.482402 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.482380    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"]
Sep 17 17:30:50.483914 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.483895    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7765f554bd-mfrns"]
Sep 17 17:30:50.529728 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.529708    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqfs7\" (UniqueName: \"kubernetes.io/projected/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-kube-api-access-tqfs7\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:30:50.529728 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.529728    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f21c7a1-40f4-4151-aab6-913b6c5f54d1-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:30:50.833841 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:50.833805    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:30:51.037874 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:51.037845    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:30:51.809724 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:30:51.809688    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f21c7a1-40f4-4151-aab6-913b6c5f54d1" path="/var/lib/kubelet/pods/1f21c7a1-40f4-4151-aab6-913b6c5f54d1/volumes"
Sep 17 17:31:55.547771 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:31:55.547724    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:32:00.693535 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:00.693487    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:32:00.875764 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:00.875726    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:32:10.424510 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:10.424462    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:32:17.259529 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:17.259495    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:32:20.589201 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:20.589159    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:32:35.982661 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:35.982571    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:32:36.014655 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.014622    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"]
Sep 17 17:32:36.015019 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.015005    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f21c7a1-40f4-4151-aab6-913b6c5f54d1" containerName="storage-initializer"
Sep 17 17:32:36.015063 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.015022    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f21c7a1-40f4-4151-aab6-913b6c5f54d1" containerName="storage-initializer"
Sep 17 17:32:36.015063 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.015035    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f21c7a1-40f4-4151-aab6-913b6c5f54d1" containerName="storage-initializer"
Sep 17 17:32:36.015063 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.015040    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f21c7a1-40f4-4151-aab6-913b6c5f54d1" containerName="storage-initializer"
Sep 17 17:32:36.015157 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.015103    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f21c7a1-40f4-4151-aab6-913b6c5f54d1" containerName="storage-initializer"
Sep 17 17:32:36.015157 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.015113    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f21c7a1-40f4-4151-aab6-913b6c5f54d1" containerName="storage-initializer"
Sep 17 17:32:36.018188 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.018168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.020843 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.020823    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:32:36.020955 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.020871    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-442a34-kube-rbac-proxy-sar-config\""
Sep 17 17:32:36.021108 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.021092    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-442a34\""
Sep 17 17:32:36.021164 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.021093    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-442a34-predictor-serving-cert\""
Sep 17 17:32:36.021164 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.021133    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:32:36.021823 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.021804    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-442a34-dockercfg-k2wkq\""
Sep 17 17:32:36.021926 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.021836    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:32:36.027215 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.027193    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"]
Sep 17 17:32:36.123966 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.123933    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kserve-provision-location\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.124118 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.123970    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-cabundle-cert\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.124118 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.124028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.124118 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.124076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-proxy-tls\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.124307 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.124154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hkfr\" (UniqueName: \"kubernetes.io/projected/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-api-access-8hkfr\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.185901 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.185873    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:32:36.224520 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.224493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kserve-provision-location\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.224626 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.224524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-cabundle-cert\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.224626 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.224546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.224626 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.224568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-proxy-tls\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.224626 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.224606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hkfr\" (UniqueName: \"kubernetes.io/projected/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-api-access-8hkfr\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.224962 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.224941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kserve-provision-location\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.225210 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.225192    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-cabundle-cert\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.225310 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.225233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.227123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.227102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-proxy-tls\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.232371 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.232351    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hkfr\" (UniqueName: \"kubernetes.io/projected/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-api-access-8hkfr\") pod \"isvc-secondary-442a34-predictor-c59998f8b-zqfmb\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") " pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.329456 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.329403    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:36.602936 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.602904    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"]
Sep 17 17:32:36.605320 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:32:36.605294    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14bfc0ad_e5d3_48ad_b6e1_3a79042cdcad.slice/crio-2adad81aba610f1cc88d9a75de0f130272c1acb055ab1ac7ed6ead382b0c60ec WatchSource:0}: Error finding container 2adad81aba610f1cc88d9a75de0f130272c1acb055ab1ac7ed6ead382b0c60ec: Status 404 returned error can't find the container with id 2adad81aba610f1cc88d9a75de0f130272c1acb055ab1ac7ed6ead382b0c60ec
Sep 17 17:32:36.947934 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.947840    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb" event={"ID":"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad","Type":"ContainerStarted","Data":"2ee535b4aa12939e681b073a5208084f19268e6488160f5e5a1615e62a5fbd86"}
Sep 17 17:32:36.947934 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:36.947911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb" event={"ID":"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad","Type":"ContainerStarted","Data":"2adad81aba610f1cc88d9a75de0f130272c1acb055ab1ac7ed6ead382b0c60ec"}
Sep 17 17:32:42.975568 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:42.975540    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-442a34-predictor-c59998f8b-zqfmb_14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad/storage-initializer/0.log"
Sep 17 17:32:42.976024 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:42.975579    2566 generic.go:356] "Generic (PLEG): container finished" podID="14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" containerID="2ee535b4aa12939e681b073a5208084f19268e6488160f5e5a1615e62a5fbd86" exitCode=1
Sep 17 17:32:42.976024 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:42.975669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb" event={"ID":"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad","Type":"ContainerDied","Data":"2ee535b4aa12939e681b073a5208084f19268e6488160f5e5a1615e62a5fbd86"}
Sep 17 17:32:43.982624 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:43.982595    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-442a34-predictor-c59998f8b-zqfmb_14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad/storage-initializer/0.log"
Sep 17 17:32:43.983000 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:43.982678    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb" event={"ID":"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad","Type":"ContainerStarted","Data":"0b0f24c5a1db5fe59ec0715be1abca9b209da7020304a4032362ff1a3c237e49"}
Sep 17 17:32:49.008067 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:49.008040    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-442a34-predictor-c59998f8b-zqfmb_14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad/storage-initializer/1.log"
Sep 17 17:32:49.008573 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:49.008462    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-442a34-predictor-c59998f8b-zqfmb_14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad/storage-initializer/0.log"
Sep 17 17:32:49.008573 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:49.008493    2566 generic.go:356] "Generic (PLEG): container finished" podID="14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" containerID="0b0f24c5a1db5fe59ec0715be1abca9b209da7020304a4032362ff1a3c237e49" exitCode=1
Sep 17 17:32:49.008573 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:49.008530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb" event={"ID":"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad","Type":"ContainerDied","Data":"0b0f24c5a1db5fe59ec0715be1abca9b209da7020304a4032362ff1a3c237e49"}
Sep 17 17:32:49.008712 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:49.008583    2566 scope.go:117] "RemoveContainer" containerID="2ee535b4aa12939e681b073a5208084f19268e6488160f5e5a1615e62a5fbd86"
Sep 17 17:32:49.008915 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:49.008897    2566 scope.go:117] "RemoveContainer" containerID="2ee535b4aa12939e681b073a5208084f19268e6488160f5e5a1615e62a5fbd86"
Sep 17 17:32:49.019062 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:32:49.019033    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-442a34-predictor-c59998f8b-zqfmb_kserve-ci-e2e-test(14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad)\"" pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb" podUID="14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad"
Sep 17 17:32:49.019421 ip-10-0-131-202 kubenswrapper[2566]: E0917 17:32:49.019386    2566 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-442a34-predictor-c59998f8b-zqfmb_kserve-ci-e2e-test_14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad_0 in pod sandbox 2adad81aba610f1cc88d9a75de0f130272c1acb055ab1ac7ed6ead382b0c60ec from index: no such id: '2ee535b4aa12939e681b073a5208084f19268e6488160f5e5a1615e62a5fbd86'" containerID="2ee535b4aa12939e681b073a5208084f19268e6488160f5e5a1615e62a5fbd86"
Sep 17 17:32:49.019488 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:49.019422    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee535b4aa12939e681b073a5208084f19268e6488160f5e5a1615e62a5fbd86"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-442a34-predictor-c59998f8b-zqfmb_kserve-ci-e2e-test_14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad_0 in pod sandbox 2adad81aba610f1cc88d9a75de0f130272c1acb055ab1ac7ed6ead382b0c60ec from index: no such id: '2ee535b4aa12939e681b073a5208084f19268e6488160f5e5a1615e62a5fbd86'"
Sep 17 17:32:50.014840 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:50.014812    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-442a34-predictor-c59998f8b-zqfmb_14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad/storage-initializer/1.log"
Sep 17 17:32:54.047386 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.047352    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"]
Sep 17 17:32:54.071684 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.071633    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:32:54.125645 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.125608    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:32:54.171577 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.171553    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-442a34-predictor-c59998f8b-zqfmb_14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad/storage-initializer/1.log"
Sep 17 17:32:54.175196 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.175180    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:54.371401 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.371326    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hkfr\" (UniqueName: \"kubernetes.io/projected/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-api-access-8hkfr\") pod \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") "
Sep 17 17:32:54.371401 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.371364    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-rbac-proxy-sar-config\") pod \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") "
Sep 17 17:32:54.371401 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.371384    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-proxy-tls\") pod \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") "
Sep 17 17:32:54.371615 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.371422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-cabundle-cert\") pod \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") "
Sep 17 17:32:54.371615 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.371475    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kserve-provision-location\") pod \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\" (UID: \"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad\") "
Sep 17 17:32:54.371713 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.371693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" (UID: "14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:32:54.371834 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.371809    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" (UID: "14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:32:54.371886 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.371811    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" (UID: "14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:32:54.373453 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.373434    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" (UID: "14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:32:54.373559 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.373521    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-api-access-8hkfr" (OuterVolumeSpecName: "kube-api-access-8hkfr") pod "14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" (UID: "14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad"). InnerVolumeSpecName "kube-api-access-8hkfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:32:54.472762 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.472740    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kserve-provision-location\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:32:54.472762 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.472760    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hkfr\" (UniqueName: \"kubernetes.io/projected/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-api-access-8hkfr\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:32:54.472888 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.472770    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:32:54.472888 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.472779    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-proxy-tls\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:32:54.472888 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:54.472787    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad-cabundle-cert\") on node \"ip-10-0-131-202.ec2.internal\" DevicePath \"\""
Sep 17 17:32:55.041754 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:55.041726    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-442a34-predictor-c59998f8b-zqfmb_14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad/storage-initializer/1.log"
Sep 17 17:32:55.041948 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:55.041820    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"
Sep 17 17:32:55.041948 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:55.041860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb" event={"ID":"14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad","Type":"ContainerDied","Data":"2adad81aba610f1cc88d9a75de0f130272c1acb055ab1ac7ed6ead382b0c60ec"}
Sep 17 17:32:55.041948 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:55.041937    2566 scope.go:117] "RemoveContainer" containerID="0b0f24c5a1db5fe59ec0715be1abca9b209da7020304a4032362ff1a3c237e49"
Sep 17 17:32:55.076588 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:55.076565    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"]
Sep 17 17:32:55.080091 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:55.080070    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-442a34-predictor-c59998f8b-zqfmb"]
Sep 17 17:32:55.808837 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:32:55.808804    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" path="/var/lib/kubelet/pods/14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad/volumes"
Sep 17 17:33:42.467507 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:33:42.467470    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:33:44.044921 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:33:44.044891    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:33:48.961461 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:33:48.961420    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:33:49.055341 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:33:49.055294    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:24.880637 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:24.880589    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:25.009926 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.009894    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrk88/must-gather-mqncd"]
Sep 17 17:34:25.010219 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.010206    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" containerName="storage-initializer"
Sep 17 17:34:25.010297 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.010220    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" containerName="storage-initializer"
Sep 17 17:34:25.010344 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.010299    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" containerName="storage-initializer"
Sep 17 17:34:25.010344 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.010307    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" containerName="storage-initializer"
Sep 17 17:34:25.010415 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.010361    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" containerName="storage-initializer"
Sep 17 17:34:25.010415 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.010367    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bfc0ad-e5d3-48ad-b6e1-3a79042cdcad" containerName="storage-initializer"
Sep 17 17:34:25.013245 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.013230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrk88/must-gather-mqncd"
Sep 17 17:34:25.015954 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.015923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lrk88\"/\"default-dockercfg-4l9gt\""
Sep 17 17:34:25.016092 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.015986    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrk88\"/\"kube-root-ca.crt\""
Sep 17 17:34:25.016896 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.016877    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrk88\"/\"openshift-service-ca.crt\""
Sep 17 17:34:25.025174 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.025155    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrk88/must-gather-mqncd"]
Sep 17 17:34:25.028705 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.028679    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:25.062360 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.062329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7215b89b-d414-4c37-acec-c9468cb6c2e0-must-gather-output\") pod \"must-gather-mqncd\" (UID: \"7215b89b-d414-4c37-acec-c9468cb6c2e0\") " pod="openshift-must-gather-lrk88/must-gather-mqncd"
Sep 17 17:34:25.062465 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.062374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd54k\" (UniqueName: \"kubernetes.io/projected/7215b89b-d414-4c37-acec-c9468cb6c2e0-kube-api-access-nd54k\") pod \"must-gather-mqncd\" (UID: \"7215b89b-d414-4c37-acec-c9468cb6c2e0\") " pod="openshift-must-gather-lrk88/must-gather-mqncd"
Sep 17 17:34:25.091858 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.091837    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:25.163294 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.163222    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd54k\" (UniqueName: \"kubernetes.io/projected/7215b89b-d414-4c37-acec-c9468cb6c2e0-kube-api-access-nd54k\") pod \"must-gather-mqncd\" (UID: \"7215b89b-d414-4c37-acec-c9468cb6c2e0\") " pod="openshift-must-gather-lrk88/must-gather-mqncd"
Sep 17 17:34:25.163374 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.163306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7215b89b-d414-4c37-acec-c9468cb6c2e0-must-gather-output\") pod \"must-gather-mqncd\" (UID: \"7215b89b-d414-4c37-acec-c9468cb6c2e0\") " pod="openshift-must-gather-lrk88/must-gather-mqncd"
Sep 17 17:34:25.163556 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.163541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7215b89b-d414-4c37-acec-c9468cb6c2e0-must-gather-output\") pod \"must-gather-mqncd\" (UID: \"7215b89b-d414-4c37-acec-c9468cb6c2e0\") " pod="openshift-must-gather-lrk88/must-gather-mqncd"
Sep 17 17:34:25.170815 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.170786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd54k\" (UniqueName: \"kubernetes.io/projected/7215b89b-d414-4c37-acec-c9468cb6c2e0-kube-api-access-nd54k\") pod \"must-gather-mqncd\" (UID: \"7215b89b-d414-4c37-acec-c9468cb6c2e0\") " pod="openshift-must-gather-lrk88/must-gather-mqncd"
Sep 17 17:34:25.329906 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.329884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrk88/must-gather-mqncd"
Sep 17 17:34:25.599621 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.599599    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrk88/must-gather-mqncd"]
Sep 17 17:34:25.601750 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:34:25.601724    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7215b89b_d414_4c37_acec_c9468cb6c2e0.slice/crio-498c1aa96d1a869b7bd6fcc55f9020284c007b5f45d7abb61edf2d0c66f5b7d3 WatchSource:0}: Error finding container 498c1aa96d1a869b7bd6fcc55f9020284c007b5f45d7abb61edf2d0c66f5b7d3: Status 404 returned error can't find the container with id 498c1aa96d1a869b7bd6fcc55f9020284c007b5f45d7abb61edf2d0c66f5b7d3
Sep 17 17:34:25.603713 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:25.603697    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:34:26.457731 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:26.457697    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrk88/must-gather-mqncd" event={"ID":"7215b89b-d414-4c37-acec-c9468cb6c2e0","Type":"ContainerStarted","Data":"498c1aa96d1a869b7bd6fcc55f9020284c007b5f45d7abb61edf2d0c66f5b7d3"}
Sep 17 17:34:27.477207 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:27.476273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrk88/must-gather-mqncd" event={"ID":"7215b89b-d414-4c37-acec-c9468cb6c2e0","Type":"ContainerStarted","Data":"7a7ee43b91cd426c49674a7e512e450a820a6fd05aba2b6a3e0e4c83a93fa382"}
Sep 17 17:34:27.477207 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:27.476327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrk88/must-gather-mqncd" event={"ID":"7215b89b-d414-4c37-acec-c9468cb6c2e0","Type":"ContainerStarted","Data":"dd1fa3fd7e9f49d98f20ba6ae8442426d48d3dce07e9541c38d2c6fab984766e"}
Sep 17 17:34:27.494483 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:27.494430    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lrk88/must-gather-mqncd" podStartSLOduration=2.62591413 podStartE2EDuration="3.494415527s" totalImagesPullingTime="868.501397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:27.493231469 +0000 UTC m=+3176.205238738" watchObservedRunningTime="2026-09-17 17:34:27.494415527 +0000 UTC m=+3176.206422858"
Sep 17 17:34:28.128635 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:28.128598    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qw97h_df7d88b0-b648-45ed-8381-e7172a404315/global-pull-secret-syncer/0.log"
Sep 17 17:34:28.194243 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:28.194212    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log"
Sep 17 17:34:28.294600 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:28.294569    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-202.ec2.internal_c990e3bd2b767d7e89440c7596b76213/haproxy/0.log"
Sep 17 17:34:30.549265 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:30.549207    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:30.615849 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:30.615799    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-264lh_7b658125-8a32-4475-a8be-5351a5064404/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:32.534070 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:32.534041    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18672a27-e8ea-4f46-a9bf-fc736c50a419/alertmanager/0.log"
Sep 17 17:34:32.566705 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:32.566672    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18672a27-e8ea-4f46-a9bf-fc736c50a419/config-reloader/0.log"
Sep 17 17:34:32.591847 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:32.591819    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18672a27-e8ea-4f46-a9bf-fc736c50a419/kube-rbac-proxy-web/0.log"
Sep 17 17:34:32.620483 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:32.620451    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18672a27-e8ea-4f46-a9bf-fc736c50a419/kube-rbac-proxy/0.log"
Sep 17 17:34:32.652138 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:32.652105    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18672a27-e8ea-4f46-a9bf-fc736c50a419/kube-rbac-proxy-metric/0.log"
Sep 17 17:34:32.683173 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:32.683140    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18672a27-e8ea-4f46-a9bf-fc736c50a419/prom-label-proxy/0.log"
Sep 17 17:34:32.717330 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:32.717291    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18672a27-e8ea-4f46-a9bf-fc736c50a419/init-config-reloader/0.log"
Sep 17 17:34:32.771894 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:32.771863    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-qd8v9_ada1cced-1b84-44be-bcf1-4dc82bed86ec/cluster-monitoring-operator/0.log"
Sep 17 17:34:33.101572 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.101533    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sntj7_20058c2c-35ba-4418-ad02-e28d14fa5aa7/node-exporter/0.log"
Sep 17 17:34:33.130347 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.130315    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sntj7_20058c2c-35ba-4418-ad02-e28d14fa5aa7/kube-rbac-proxy/0.log"
Sep 17 17:34:33.158939 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.158906    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sntj7_20058c2c-35ba-4418-ad02-e28d14fa5aa7/init-textfile/0.log"
Sep 17 17:34:33.286773 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.286740    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-hllw6_c6703ecf-d009-4e17-9503-39272491d079/kube-rbac-proxy-main/0.log"
Sep 17 17:34:33.321361 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.321330    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-hllw6_c6703ecf-d009-4e17-9503-39272491d079/kube-rbac-proxy-self/0.log"
Sep 17 17:34:33.350441 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.350408    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-hllw6_c6703ecf-d009-4e17-9503-39272491d079/openshift-state-metrics/0.log"
Sep 17 17:34:33.601104 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.601067    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-2wlcg_943ec8cb-97a1-4399-9f17-7860e6a22e3f/prometheus-operator/0.log"
Sep 17 17:34:33.636302 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.636275    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-2wlcg_943ec8cb-97a1-4399-9f17-7860e6a22e3f/kube-rbac-proxy/0.log"
Sep 17 17:34:33.678066 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.678041    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-jlh62_01ade8bd-2c86-442f-8652-1d14b67163c5/prometheus-operator-admission-webhook/0.log"
Sep 17 17:34:33.720109 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.720083    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8846c548-t74j2_eabba0a4-769e-47f9-a74a-55dbf9a1fe5c/telemeter-client/0.log"
Sep 17 17:34:33.755514 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.755444    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8846c548-t74j2_eabba0a4-769e-47f9-a74a-55dbf9a1fe5c/reload/0.log"
Sep 17 17:34:33.793956 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:33.793919    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8846c548-t74j2_eabba0a4-769e-47f9-a74a-55dbf9a1fe5c/kube-rbac-proxy/0.log"
Sep 17 17:34:35.119861 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:35.119833    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-fgjdr_42e07a4b-4c5d-4099-ad81-a7a58dccc1b8/networking-console-plugin/0.log"
Sep 17 17:34:35.995077 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:35.995047    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-546bb6cdb4-bjfrg_9a3584ac-7057-4dc8-a9ab-570b6833b830/console/0.log"
Sep 17 17:34:36.018283 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.018233    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"]
Sep 17 17:34:36.024042 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.024016    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.030193 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.030156    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"]
Sep 17 17:34:36.178369 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.178333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2rgq\" (UniqueName: \"kubernetes.io/projected/5953b900-587e-4e9d-a2a5-664ed46cf90e-kube-api-access-n2rgq\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.178740 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.178383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-sys\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.178740 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.178418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-lib-modules\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.178740 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.178451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-proc\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.178740 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.178473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-podres\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.279939 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.279844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rgq\" (UniqueName: \"kubernetes.io/projected/5953b900-587e-4e9d-a2a5-664ed46cf90e-kube-api-access-n2rgq\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.279939 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.279917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-sys\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.280172 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.279966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-lib-modules\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.280172 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.279992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-proc\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.280172 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.280025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-podres\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.280380 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.280214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-podres\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.280632 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.280607    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-sys\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.280748 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.280730    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-lib-modules\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.280807 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.280787    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5953b900-587e-4e9d-a2a5-664ed46cf90e-proc\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.288275 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.288229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rgq\" (UniqueName: \"kubernetes.io/projected/5953b900-587e-4e9d-a2a5-664ed46cf90e-kube-api-access-n2rgq\") pod \"perf-node-gather-daemonset-cvp4m\" (UID: \"5953b900-587e-4e9d-a2a5-664ed46cf90e\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.336330 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.336296    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:36.626565 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:36.626477    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"]
Sep 17 17:34:36.631338 ip-10-0-131-202 kubenswrapper[2566]: W0917 17:34:36.631304    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5953b900_587e_4e9d_a2a5_664ed46cf90e.slice/crio-80f315b6867910b40a1305baa8f1c0ade461e36e8a9b96a7c49fedf3b68eb298 WatchSource:0}: Error finding container 80f315b6867910b40a1305baa8f1c0ade461e36e8a9b96a7c49fedf3b68eb298: Status 404 returned error can't find the container with id 80f315b6867910b40a1305baa8f1c0ade461e36e8a9b96a7c49fedf3b68eb298
Sep 17 17:34:37.448123 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:37.448093    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9kh8p_fde1bd0b-5754-49fd-84c1-28f2383539aa/dns/0.log"
Sep 17 17:34:37.468205 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:37.468179    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9kh8p_fde1bd0b-5754-49fd-84c1-28f2383539aa/kube-rbac-proxy/0.log"
Sep 17 17:34:37.538550 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:37.538512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m" event={"ID":"5953b900-587e-4e9d-a2a5-664ed46cf90e","Type":"ContainerStarted","Data":"b378981726b86523c4b977a5c40cf1cde92696b980be37dc1bb5d0733db0c404"}
Sep 17 17:34:37.538713 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:37.538571    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m" event={"ID":"5953b900-587e-4e9d-a2a5-664ed46cf90e","Type":"ContainerStarted","Data":"80f315b6867910b40a1305baa8f1c0ade461e36e8a9b96a7c49fedf3b68eb298"}
Sep 17 17:34:37.538713 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:37.538599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:37.553049 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:37.553002    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m" podStartSLOduration=1.552987565 podStartE2EDuration="1.552987565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:37.552514621 +0000 UTC m=+3186.264521891" watchObservedRunningTime="2026-09-17 17:34:37.552987565 +0000 UTC m=+3186.264994832"
Sep 17 17:34:37.604339 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:37.604311    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8hvh7_55dd2af7-8fe9-4f9d-aad5-680fdeb3dfbe/dns-node-resolver/0.log"
Sep 17 17:34:38.079021 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:38.078985    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-674f4bb9d7-lq5lr_d2e287a7-f9ee-4fe6-86c9-a987d3485c1c/registry/0.log"
Sep 17 17:34:38.142979 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:38.142950    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h774k_89d257ca-2fd9-45a8-b0a3-7cfff6e8cd10/node-ca/0.log"
Sep 17 17:34:38.894332 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:38.894298    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b7d668668-7nwdq_9b164a3b-fc9f-48c7-a6ac-c2e3012379b8/router/0.log"
Sep 17 17:34:39.255125 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:39.255098    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-md9qz_be9dad4e-fb00-4e83-bbf1-5cb7614d1e3b/serve-healthcheck-canary/0.log"
Sep 17 17:34:39.787627 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:39.787594    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qzw8z_dd5495c6-27d1-4d2f-a513-52453bc60ca2/kube-rbac-proxy/0.log"
Sep 17 17:34:39.806571 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:39.806537    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qzw8z_dd5495c6-27d1-4d2f-a513-52453bc60ca2/exporter/0.log"
Sep 17 17:34:39.827563 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:39.827521    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qzw8z_dd5495c6-27d1-4d2f-a513-52453bc60ca2/extractor/0.log"
Sep 17 17:34:42.208934 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:42.208897    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-846cf5d849-8t4bs_d6f61d2d-0246-4dd1-8baa-b8eb1603845d/manager/0.log"
Sep 17 17:34:42.267031 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:42.267009    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-zlbmp_414c5dd5-a07a-4609-a58a-c3fa34686fd2/server/0.log"
Sep 17 17:34:42.823198 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:42.823166    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-974pg_0fca3dd1-cd4f-4cee-a6e6-b39170c9c78e/manager/0.log"
Sep 17 17:34:42.869892 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:42.869862    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-65jpf_9deb91f9-9168-4b0e-9e7a-4a18760d377b/s3-init/0.log"
Sep 17 17:34:42.887442 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:42.887419    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-65jpf_9deb91f9-9168-4b0e-9e7a-4a18760d377b/download-hf-model/0.log"
Sep 17 17:34:42.920990 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:42.920964    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-cjbq7_11acc44c-aa4f-4098-8dc7-3d988d403efb/s3-tls-init-custom/0.log"
Sep 17 17:34:42.936423 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:42.936398    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-cjbq7_11acc44c-aa4f-4098-8dc7-3d988d403efb/download-hf-model/0.log"
Sep 17 17:34:42.966723 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:42.966703    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-xbt6x_d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06/s3-tls-init-serving/0.log"
Sep 17 17:34:42.985825 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:42.985800    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-xbt6x_d1f8e6b0-86ed-4a28-a53f-beeb06f7cd06/download-hf-model/0.log"
Sep 17 17:34:43.037348 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:43.037329    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-lsrbh_0ee6c2c0-9d1e-4ef9-b93b-a562d4201359/seaweedfs-tls-custom/0.log"
Sep 17 17:34:43.064526 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:43.064502    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-kdbhx_34024e5e-282f-42cd-9f6e-24124ffe129c/seaweedfs-tls-serving/0.log"
Sep 17 17:34:43.559221 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:43.559190    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-cvp4m"
Sep 17 17:34:48.007605 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:48.007571    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-bl8n9_3efa2ff4-6070-46a8-941d-0c42ee9f24e1/kube-storage-version-migrator-operator/1.log"
Sep 17 17:34:48.009633 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:48.009595    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-bl8n9_3efa2ff4-6070-46a8-941d-0c42ee9f24e1/kube-storage-version-migrator-operator/0.log"
Sep 17 17:34:49.095596 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:49.095569    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkvgc_48abb272-fae7-4755-a74e-f82be2c3b6f0/kube-multus-additional-cni-plugins/0.log"
Sep 17 17:34:49.120184 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:49.120160    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkvgc_48abb272-fae7-4755-a74e-f82be2c3b6f0/egress-router-binary-copy/0.log"
Sep 17 17:34:49.144281 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:49.144248    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkvgc_48abb272-fae7-4755-a74e-f82be2c3b6f0/cni-plugins/0.log"
Sep 17 17:34:49.166352 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:49.166333    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkvgc_48abb272-fae7-4755-a74e-f82be2c3b6f0/bond-cni-plugin/0.log"
Sep 17 17:34:49.191632 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:49.191612    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkvgc_48abb272-fae7-4755-a74e-f82be2c3b6f0/routeoverride-cni/0.log"
Sep 17 17:34:49.215898 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:49.215873    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkvgc_48abb272-fae7-4755-a74e-f82be2c3b6f0/whereabouts-cni-bincopy/0.log"
Sep 17 17:34:49.240085 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:49.240066    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkvgc_48abb272-fae7-4755-a74e-f82be2c3b6f0/whereabouts-cni/0.log"
Sep 17 17:34:49.546097 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:49.546050    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x4282_dce7e7cb-88b0-4419-b2c9-144f94438855/kube-multus/0.log"
Sep 17 17:34:49.674606 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:49.674578    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vm27c_4efac150-5db6-42c1-8309-138120e7f820/network-metrics-daemon/0.log"
Sep 17 17:34:49.691399 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:49.691375    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vm27c_4efac150-5db6-42c1-8309-138120e7f820/kube-rbac-proxy/0.log"
Sep 17 17:34:50.889737 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:50.889702    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxtcx_c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b/ovn-controller/0.log"
Sep 17 17:34:50.908004 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:50.907974    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxtcx_c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b/ovn-acl-logging/0.log"
Sep 17 17:34:50.934760 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:50.934734    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxtcx_c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b/kube-rbac-proxy-node/0.log"
Sep 17 17:34:50.956321 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:50.956295    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxtcx_c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 17:34:50.975387 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:50.975361    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxtcx_c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b/northd/0.log"
Sep 17 17:34:50.997623 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:50.997599    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxtcx_c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b/nbdb/0.log"
Sep 17 17:34:51.019657 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:51.019631    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxtcx_c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b/sbdb/0.log"
Sep 17 17:34:51.086259 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:51.086234    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cxtcx_c3a6b9e3-1b73-4a0e-b75d-1c4c8e81314b/ovnkube-controller/0.log"
Sep 17 17:34:52.126989 ip-10-0-131-202 kubenswrapper[2566]: I0917 17:34:52.126960    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-w6lnl_8a1950af-8604-4048-93b4-1bdb996f291e/network-check-target-container/0.log"