Jul 28 17:17:13.871377 ip-10-0-132-11 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:17:14.263276 ip-10-0-132-11 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:17:14.263276 ip-10-0-132-11 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:17:14.263276 ip-10-0-132-11 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:17:14.263276 ip-10-0-132-11 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:17:14.263276 ip-10-0-132-11 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:17:14.263936 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.263268    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:17:14.267981 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.267946    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:14.267981 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.267982    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.267990    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.267995    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268000    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268005    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268010    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268015    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268033    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268037    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268042    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268048    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268054    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268065    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268069    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268073    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268077    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268081    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268086    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268090    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268094    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:14.268158 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268098    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268102    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268107    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268111    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268115    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268124    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268128    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268132    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268136    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268141    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268146    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268150    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268154    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268158    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268163    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268222    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268237    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268243    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268249    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268254    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:14.268731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268258    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268264    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268268    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268272    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268277    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268281    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268292    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268296    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268301    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268305    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268310    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268314    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268318    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268322    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268329    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268341    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268346    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268351    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268355    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268373    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:14.269245 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268377    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268401    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268417    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268420    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268527    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268531    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268534    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268537    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268541    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268548    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268551    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268555    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268558    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268561    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268564    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268567    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268570    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268573    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268576    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:14.269727 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268579    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268581    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268584    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268586    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268589    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268592    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268594    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268597    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268599    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268602    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268604    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268607    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268609    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268612    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268614    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268617    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268621    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268624    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.268626    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269352    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:14.270248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269359    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269363    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269366    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269369    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269372    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269375    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269377    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269380    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269383    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269385    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269388    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269390    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269392    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269395    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269400    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269403    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269406    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269409    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269412    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:14.270820 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269414    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269417    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269419    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269422    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269425    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269428    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269430    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269433    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269436    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269439    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269442    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269444    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269447    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269449    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269454    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269457    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269460    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269463    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269465    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269468    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:14.271315 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269470    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269474    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269476    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269480    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269484    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269487    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269490    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269493    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269495    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269498    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269501    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269505    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269508    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269510    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269513    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269515    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269518    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269520    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269523    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269525    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:14.271806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269528    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269530    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269533    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269535    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269538    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269540    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269544    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269547    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269549    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269552    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269555    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269557    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269560    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269562    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269565    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269567    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269573    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269576    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269579    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:14.272304 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269581    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269584    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269586    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269589    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269592    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269595    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269597    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269600    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269603    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269605    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269608    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269610    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269613    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269615    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269618    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269621    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269624    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269626    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269628    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.269632    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:14.272921 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270099    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270106    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270109    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270112    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270115    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270118    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270120    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270123    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270125    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270128    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270131    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270134    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270136    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270139    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270142    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270145    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270149    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270151    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270154    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:14.273806 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270157    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270159    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270162    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270164    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270167    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270170    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270174    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270177    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270180    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270182    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270185    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270187    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270193    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270195    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270198    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270200    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270203    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270206    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270208    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:14.274359 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270211    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270214    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270217    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270219    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270222    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270224    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270227    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270230    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270233    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270236    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270239    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270242    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270244    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270247    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270249    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270252    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270254    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270257    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270260    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:14.274860 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270262    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270265    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270267    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270271    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270274    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270277    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270281    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270284    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270287    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270290    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270292    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270295    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270297    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270300    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270302    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270304    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270307    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270309    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270313    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270315    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:14.275387 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270317    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270321    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270323    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270326    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270329    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270332    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270334    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270337    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270339    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270342    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270344    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270347    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270349    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270352    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270354    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270356    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270359    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270361    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270364    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270367    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:14.275965 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270369    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.270372    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270441    2583 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270448    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270455    2583 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270459    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270463    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270467    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270473    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270477    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270481    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270484    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270487    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270490    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270493    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270496    2583 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270499    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270503    2583 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270506    2583 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270509    2583 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270513    2583 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270516    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270519    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270522    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:17:14.276504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270525    2583 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270529    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270532    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270535    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270538    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270542    2583 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270545    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270548    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270551    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270554    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270558    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270561    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270564    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270567    2583 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270571    2583 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270573    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270577    2583 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270581    2583 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270583    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270586    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270589    2583 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270593    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270596    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270599    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270602    2583 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:17:14.277144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270605    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270607    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270610    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270613    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270616    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270619    2583 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270622    2583 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270626    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270629    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270631    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270634    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270637    2583 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270640    2583 flags.go:64] FLAG: --help="false"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270643    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270646    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270649    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270652    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270655    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270658    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270661    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270664    2583 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270668    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270670    2583 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:17:14.277806 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270673    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270676    2583 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270679    2583 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270682    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270684    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270687    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270690    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270693    2583 flags.go:64] FLAG: --lock-file=""
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270695    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270698    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270701    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270706    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270710    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270713    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270716    2583 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270719    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270722    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270725    2583 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270727    2583 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270732    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270735    2583 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270739    2583 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270744    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270747    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270750    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:17:14.278525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270753    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270756    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270759    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270762    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270770    2583 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270773    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270776    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270779    2583 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270782    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270788    2583 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270791    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270794    2583 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270797    2583 flags.go:64] FLAG: --port="10250"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270800    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270803    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0be4e3cbdfb74e35c"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270807    2583 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270810    2583 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270813    2583 flags.go:64] FLAG: --register-node="true"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270816    2583 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270819    2583 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270823    2583 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270826    2583 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270829    2583 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270833    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270836    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:17:14.279311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270839    2583 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270842    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270845    2583 flags.go:64] FLAG: --runonce="false"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270848    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270852    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270856    2583 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270859    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270863    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270866    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270869    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270872    2583 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270875    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270878    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270881    2583 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270884    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270887    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270890    2583 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270893    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270898    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270901    2583 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270904    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270912    2583 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270915    2583 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270918    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270921    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:17:14.280004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270925    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270927    2583 flags.go:64] FLAG: --v="2"
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270932    2583 flags.go:64] FLAG: --version="false"
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270937    2583 flags.go:64] FLAG: --vmodule=""
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270941    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.270944    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271088    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271093    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271096    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271099    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271102    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271104    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271108    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271111    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271115    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271117    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271120    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271123    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271125    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271128    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271131    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:14.280688 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271134    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271137    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271139    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271142    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271145    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271148    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271150    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271153    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271155    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271158    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271161    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271164    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271166    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271169    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271171    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271174    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271177    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271181    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271184    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:14.281287 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271188    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271190    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271193    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271196    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271200    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271203    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271206    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271209    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271212    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271214    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271217    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271220    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271222    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271225    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271228    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271230    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271232    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271235    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271238    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271240    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:14.281793 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271243    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271245    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271248    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271250    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271253    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271256    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271258    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271261    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271263    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271267    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271271    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271274    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271277    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271280    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271283    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271285    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271289    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271292    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271299    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:14.282392 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271303    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271305    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271308    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271310    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271313    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271316    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271319    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271322    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271325    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271327    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271330    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271333    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271335    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271338    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271340    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271343    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271346    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271348    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271351    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271353    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:14.282895 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271356    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271358    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271361    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271365    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271367    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.271370    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.272352    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.279592    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.279609    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.279632    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.279639    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279699    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279707    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279711    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:14.283458 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279714    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279718    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279721    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279724    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279728    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279731    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279734    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279737    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279740    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279743    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279746    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279751    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279755    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279758    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279763    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279767    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279770    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279773    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279776    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:14.283844 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279779    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279782    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279784    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279787    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279790    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279793    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279796    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279801    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279804    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279807    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279809    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279812    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279815    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279818    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279821    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279824    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279827    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279829    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279832    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:14.284363 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279835    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279838    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279840    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279843    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279846    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279849    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279852    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279855    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279857    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279860    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279863    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279866    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279868    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279872    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279875    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279878    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279880    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279883    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279886    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279889    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:14.284851 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279893    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279896    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279899    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279902    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279905    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279907    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279910    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279913    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279915    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279918    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279921    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279923    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279926    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279929    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279932    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279935    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279937    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279940    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279943    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279946    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279949    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:14.285386 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279951    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279954    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279957    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279960    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279962    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279965    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279968    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279971    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279973    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279976    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279979    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279983    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279986    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279989    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.279992    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280001    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:14.285964 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280003    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.280009    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280172    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280180    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280184    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280188    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280191    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280195    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280197    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280200    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280204    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280206    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280209    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280212    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280215    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:14.286486 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280218    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280221    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280223    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280226    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280229    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280232    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280235    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280237    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280241    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280243    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280246    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280249    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280252    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280255    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280258    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280261    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280264    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280267    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280270    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280274    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:14.286896 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280278    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280282    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280284    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280287    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280290    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280293    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280295    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280298    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280301    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280303    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280306    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280309    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280312    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280314    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280317    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280320    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280322    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280325    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280328    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:14.287440 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280331    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280333    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280336    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280339    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280342    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280345    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280348    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280350    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280353    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280356    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280359    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280362    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280364    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280367    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280370    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280373    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280376    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280379    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280382    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280384    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:14.287960 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280387    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280390    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280393    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280395    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280398    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280401    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280403    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280406    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280409    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280412    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280414    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280417    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280420    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280422    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280425    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280428    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280431    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280434    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280436    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280439    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:14.288493 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280442    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280446    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280454    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280457    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280460    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280463    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:14.280467    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.280471    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.280603    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.282787    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.284608    2583 server.go:1019] "Starting client certificate rotation"
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.284720    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.285837    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:17:14.289022 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.287678    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:17:14.289383 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.288908    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:17:14.301539 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.301510    2583 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:17:14.303990 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.303970    2583 log.go:25] "Validated CRI v1 image API"
Jul 28 17:17:14.305668 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.305640    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:17:14.308242 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.308219    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 96b3b10f-0054-4c45-9c3c-b6b6dac15c8d:/dev/nvme0n1p4 fb5baa14-792b-424e-9dc0-64e0d2bf5014:/dev/nvme0n1p3]
Jul 28 17:17:14.308390 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.308241    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:17:14.315392 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.315255    2583 manager.go:216] Machine: {Timestamp:2026-07-28 17:17:14.313730462 +0000 UTC m=+0.327110349 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098257 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e9ef6d326fa958933c15565de9745 SystemUUID:ec2e9ef6-d326-fa95-8933-c15565de9745 BootID:f703103e-cfac-4819-8be2-f4676718ff51 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:94:a2:ec:6f:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:94:a2:ec:6f:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:0d:2c:63:15:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:17:14.315392 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.315373    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:17:14.315575 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.315556    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:17:14.318705 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.318657    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:17:14.318788 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.318685    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:17:14.318925 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.318709    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-11.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:17:14.319594 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.319581    2583 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:17:14.319659 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.319601    2583 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:17:14.319659 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.319613    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:17:14.320221 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.320209    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:17:14.321121 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.321109    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:17:14.323514 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.323499    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:17:14.323662 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.323650    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:17:14.323721 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.323692    2583 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:17:14.323721 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.323709    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:17:14.323823 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.323729    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:17:14.323823 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.323744    2583 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:17:14.323823 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.323758    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:17:14.324189 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.324174    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:17:14.324258 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.324199    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:17:14.326753 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.326732    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:17:14.328264 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.328249    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:17:14.328321 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.328283    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:17:14.329811 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329796    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:17:14.329862 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329820    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:17:14.329862 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329828    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:17:14.329862 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329834    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:17:14.329862 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329843    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:17:14.329862 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329849    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:17:14.329862 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329857    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:17:14.329862 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329863    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:17:14.330073 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329870    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:17:14.330073 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329876    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:17:14.330073 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329886    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:17:14.330073 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.329891    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:17:14.330537 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.330525    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:17:14.330537 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.330536    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:17:14.334273 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.334254    2583 server.go:1268] "Started kubelet"
Jul 28 17:17:14.334505 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.334416    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:17:14.334615 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.334503    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:17:14.334615 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.334541    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:17:14.338189 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.338156    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:17:14.338251 ip-10-0-132-11 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:17:14.340721 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.339661    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-11.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:17:14.340721 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.339691    2583 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:17:14.340721 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.339989    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:17:14.340721 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.340309    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:17:14.341940 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.341921    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:17:14.344191 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.344172    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q7bh2"
Jul 28 17:17:14.349045 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.349020    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:17:14.349163 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.349049    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:17:14.349352 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.349311    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-11.ec2.internal\" not found"
Jul 28 17:17:14.349767 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.349745    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:17:14.349767 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.349769    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:17:14.349911 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.349814    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:17:14.349965 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.349937    2583 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:17:14.349965 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.349947    2583 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:17:14.350301 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.350277    2583 factory.go:55] Registering systemd factory
Jul 28 17:17:14.350301 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.350300    2583 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:17:14.350603 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.350584    2583 factory.go:153] Registering CRI-O factory
Jul 28 17:17:14.350688 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.350607    2583 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:17:14.350751 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.350701    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:17:14.350751 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.350728    2583 factory.go:103] Registering Raw factory
Jul 28 17:17:14.350751 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.350746    2583 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:17:14.351174 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.351157    2583 manager.go:349] Starting recovery of all containers
Jul 28 17:17:14.351490 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.351455    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:17:14.352296 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.352250    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:17:14.352586 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.351622    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-11.ec2.internal.18c683a20d8c5a6d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-11.ec2.internal,UID:ip-10-0-132-11.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-11.ec2.internal,},FirstTimestamp:2026-07-28 17:17:14.334222957 +0000 UTC m=+0.347602844,LastTimestamp:2026-07-28 17:17:14.334222957 +0000 UTC m=+0.347602844,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-11.ec2.internal,}"
Jul 28 17:17:14.353250 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.353220    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q7bh2"
Jul 28 17:17:14.360565 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.360542    2583 manager.go:354] Recovery completed
Jul 28 17:17:14.365633 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.365619    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:14.375926 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.375906    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:14.376009 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.375939    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:14.376009 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.375951    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:14.376562 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.376543    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:17:14.376562 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.376561    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:17:14.376679 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.376571    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:14.376679 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.376581    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:17:14.377461 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.377441    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:14.377553 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.377477    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:14.377553 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.377492    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:14.378528 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.378515    2583 policy_none.go:49] "None policy: Start"
Jul 28 17:17:14.378583 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.378532    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:17:14.378583 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.378544    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:17:14.379961 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.379949    2583 policy_none.go:47] "Start"
Jul 28 17:17:14.418929 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.418709    2583 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:17:14.419051 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.418955    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:17:14.419051 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.418969    2583 server.go:87] "Starting device plugin registration server"
Jul 28 17:17:14.419244 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.419232    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:17:14.419282 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.419246    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:17:14.419390 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.419359    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:17:14.419512 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.419467    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:17:14.419512 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.419477    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:17:14.420089 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.420066    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:17:14.420163 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.420133    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-11.ec2.internal\" not found"
Jul 28 17:17:14.519769 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.519659    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:14.521091 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.520792    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:14.521091 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.520826    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:14.521091 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.520844    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:14.521091 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.520882    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.526931 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.526912    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.526984 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.526940    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-11.ec2.internal\": node \"ip-10-0-132-11.ec2.internal\" not found"
Jul 28 17:17:14.529733 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.529691    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:17:14.531178 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.531155    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:17:14.531262 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.531205    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:17:14.531262 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.531233    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:17:14.531345 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.531277    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:17:14.533894 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.533874    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:14.545324 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.545304    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-11.ec2.internal\" not found"
Jul 28 17:17:14.632320 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.632262    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-11.ec2.internal"]
Jul 28 17:17:14.632467 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.632367    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:14.633540 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.633514    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:14.633650 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.633553    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:14.633650 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.633565    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:14.635261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.635244    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:14.635457 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.635440    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.635526 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.635481    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:14.636238 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.636219    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:14.636343 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.636251    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:14.636343 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.636263    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:14.636343 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.636250    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:14.636460 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.636361    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:14.636460 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.636376    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:14.637750 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.637733    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.637799 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.637768    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:14.638612 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.638597    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:14.638705 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.638622    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:14.638705 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.638633    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:14.646152 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.646136    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-11.ec2.internal\" not found"
Jul 28 17:17:14.650445 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.650427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29924a4ae605b9b2aa658a2296cc479f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal\" (UID: \"29924a4ae605b9b2aa658a2296cc479f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.650515 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.650453    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29924a4ae605b9b2aa658a2296cc479f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal\" (UID: \"29924a4ae605b9b2aa658a2296cc479f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.650515 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.650468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8cc528b9c87d3a6a58c64430cacff3d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-11.ec2.internal\" (UID: \"a8cc528b9c87d3a6a58c64430cacff3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.666116 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.666086    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-11.ec2.internal\" not found" node="ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.671216 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.671196    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-11.ec2.internal\" not found" node="ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.747066 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.747017    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-11.ec2.internal\" not found"
Jul 28 17:17:14.751214 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.751192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8cc528b9c87d3a6a58c64430cacff3d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-11.ec2.internal\" (UID: \"a8cc528b9c87d3a6a58c64430cacff3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.751296 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.751221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29924a4ae605b9b2aa658a2296cc479f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal\" (UID: \"29924a4ae605b9b2aa658a2296cc479f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.751296 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.751241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29924a4ae605b9b2aa658a2296cc479f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal\" (UID: \"29924a4ae605b9b2aa658a2296cc479f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.751296 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.751286    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29924a4ae605b9b2aa658a2296cc479f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal\" (UID: \"29924a4ae605b9b2aa658a2296cc479f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.751408 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.751313    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8cc528b9c87d3a6a58c64430cacff3d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-11.ec2.internal\" (UID: \"a8cc528b9c87d3a6a58c64430cacff3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.751408 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.751317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29924a4ae605b9b2aa658a2296cc479f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal\" (UID: \"29924a4ae605b9b2aa658a2296cc479f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.848144 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.848056    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-11.ec2.internal\" not found"
Jul 28 17:17:14.948929 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:14.948885    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-11.ec2.internal\" not found"
Jul 28 17:17:14.968615 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.968583    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:14.974022 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:14.974005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:15.049664 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.049605    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-11.ec2.internal\" not found"
Jul 28 17:17:15.157244 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.157135    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-11.ec2.internal\" not found"
Jul 28 17:17:15.160250 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.160231    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:15.255290 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.255249    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:15.269115 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.269088    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:17:15.270878 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.270861    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-11.ec2.internal"
Jul 28 17:17:15.279501 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.279477    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:17:15.283910 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.283893    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:17:15.284124 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.284076    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:17:15.284124 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.284080    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:17:15.324729 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.324699    2583 apiserver.go:52] "Watching apiserver"
Jul 28 17:17:15.340937 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.340902    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:17:15.342565 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.342523    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-bsr6p","openshift-network-diagnostics/network-check-target-qllgl","kube-system/konnectivity-agent-6vmg2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk","openshift-dns/node-resolver-hbgp6","openshift-image-registry/node-ca-sm9zr","openshift-multus/multus-additional-cni-plugins-5kr5s","openshift-multus/multus-dzwxf","openshift-network-operator/iptables-alerter-pb79v","openshift-ovn-kubernetes/ovnkube-node-mdz9h","kube-system/global-pull-secret-syncer-56htm","kube-system/kube-apiserver-proxy-ip-10-0-132-11.ec2.internal","openshift-cluster-node-tuning-operator/tuned-lr49l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal"]
Jul 28 17:17:15.344635 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.344604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:15.344744 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.344719    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:15.345963 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.345946    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:15.346031 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.346013    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:15.347329 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.347311    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:15.348472 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.348454    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.349187 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.349171    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:17:15.349721 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.349699    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:17:15.349849 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.349702    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hbgp6"
Jul 28 17:17:15.350211 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.350187    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x2pz5\""
Jul 28 17:17:15.350307 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.350230    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:17:15.350865 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.350847    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sm9zr"
Jul 28 17:17:15.350956 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.350875    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:17:15.350956 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.350926    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:17:15.351146 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.350998    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8q9rl\""
Jul 28 17:17:15.351146 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.351001    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:17:15.352032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.352016    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:17:15.352324 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.352310    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jk4jv\""
Jul 28 17:17:15.353192 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.353177    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:17:15.353305 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.353290    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.353380 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.353353    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:17:15.353592 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.353577    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:17:15.354062 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.354033    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-87wbm\""
Jul 28 17:17:15.354062 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.354040    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:17:15.354479 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.354466    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.355767 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.355753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pb79v"
Jul 28 17:17:15.356340 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.356313    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:17:15.356439 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.356365    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:17:15.356439 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.356392    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:17:15.356632 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.356617    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2q9s9\""
Jul 28 17:17:15.356632 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.356629    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:17:15.356727 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.356712    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:17:15.356773 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.356753    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:17:15.357080 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357067    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fffnx\""
Jul 28 17:17:15.357209 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357186    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.357751 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.357833 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-registration-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.357833 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357797    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29111c58-63e9-4233-a4ef-0983db2a6906-hosts-file\") pod \"node-resolver-hbgp6\" (UID: \"29111c58-63e9-4233-a4ef-0983db2a6906\") " pod="openshift-dns/node-resolver-hbgp6"
Jul 28 17:17:15.357833 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29111c58-63e9-4233-a4ef-0983db2a6906-tmp-dir\") pod \"node-resolver-hbgp6\" (UID: \"29111c58-63e9-4233-a4ef-0983db2a6906\") " pod="openshift-dns/node-resolver-hbgp6"
Jul 28 17:17:15.357974 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-device-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.357974 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357899    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drgr7\" (UniqueName: \"kubernetes.io/projected/ae56b753-916d-4422-a983-1f87ffc43ec6-kube-api-access-drgr7\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.357974 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xnwn\" (UniqueName: \"kubernetes.io/projected/37cc1108-c664-4196-8d56-259edfb151d5-kube-api-access-9xnwn\") pod \"node-ca-sm9zr\" (UID: \"37cc1108-c664-4196-8d56-259edfb151d5\") " pod="openshift-image-registry/node-ca-sm9zr"
Jul 28 17:17:15.357974 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357679    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:12:15 +0000 UTC" deadline="2028-01-16 17:23:15.46023802 +0000 UTC"
Jul 28 17:17:15.357974 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:15.358196 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37cc1108-c664-4196-8d56-259edfb151d5-host\") pod \"node-ca-sm9zr\" (UID: \"37cc1108-c664-4196-8d56-259edfb151d5\") " pod="openshift-image-registry/node-ca-sm9zr"
Jul 28 17:17:15.358196 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-sys-fs\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.358196 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.357975    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12888h6m0.102269061s"
Jul 28 17:17:15.358196 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358052    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx6z4\" (UniqueName: \"kubernetes.io/projected/29111c58-63e9-4233-a4ef-0983db2a6906-kube-api-access-fx6z4\") pod \"node-resolver-hbgp6\" (UID: \"29111c58-63e9-4233-a4ef-0983db2a6906\") " pod="openshift-dns/node-resolver-hbgp6"
Jul 28 17:17:15.358196 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6vsg\" (UniqueName: \"kubernetes.io/projected/67fc1b7f-80b5-41d9-9141-9bcea29579d4-kube-api-access-r6vsg\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:15.358196 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358096    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67mg\" (UniqueName: \"kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg\") pod \"network-check-target-qllgl\" (UID: \"147f1d2b-fe6a-4336-a15c-6362fc89ec8f\") " pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:15.358196 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358116    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7-agent-certs\") pod \"konnectivity-agent-6vmg2\" (UID: \"ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7\") " pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:15.358196 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7-konnectivity-ca\") pod \"konnectivity-agent-6vmg2\" (UID: \"ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7\") " pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:15.358543 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-socket-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.358543 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-etc-selinux\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.358543 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358242    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:15.358543 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37cc1108-c664-4196-8d56-259edfb151d5-serviceca\") pod \"node-ca-sm9zr\" (UID: \"37cc1108-c664-4196-8d56-259edfb151d5\") " pod="openshift-image-registry/node-ca-sm9zr"
Jul 28 17:17:15.358543 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358300    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:15.358741 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358646    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:15.358741 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.358652    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vdc57\""
Jul 28 17:17:15.358741 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.358701    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:15.359098 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.359058    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:17:15.360051 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.360032    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.360211 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.360187    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:17:15.360470 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.360447    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:17:15.360553 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.360471    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:17:15.360655 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.360638    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:17:15.361325 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.361083    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:17:15.361325 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.361183    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sjjq2\""
Jul 28 17:17:15.361325 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.361284    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:17:15.362764 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.362549    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:15.362764 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.362680    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:15.363622 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.362955    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wwlbx\""
Jul 28 17:17:15.365057 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.365034    2583 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded18bab3_8bf4_4663_8f75_1f3a6fe2daf7.slice/cpuset.cpus.effective": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded18bab3_8bf4_4663_8f75_1f3a6fe2daf7.slice/cpuset.cpus.effective: no such device
Jul 28 17:17:15.365960 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.365941    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:17:15.387921 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.387897    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6tc4z"
Jul 28 17:17:15.404221 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.404198    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6tc4z"
Jul 28 17:17:15.450664 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.450637    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:17:15.458577 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-run-netns\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.458692 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-env-overrides\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.458692 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458620    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-sysconfig\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.458692 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-system-cni-dir\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.458692 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx6z4\" (UniqueName: \"kubernetes.io/projected/29111c58-63e9-4233-a4ef-0983db2a6906-kube-api-access-fx6z4\") pod \"node-resolver-hbgp6\" (UID: \"29111c58-63e9-4233-a4ef-0983db2a6906\") " pod="openshift-dns/node-resolver-hbgp6"
Jul 28 17:17:15.458941 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7-agent-certs\") pod \"konnectivity-agent-6vmg2\" (UID: \"ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7\") " pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:15.458941 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-systemd-units\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.458941 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-node-log\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.458941 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-modprobe-d\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.458941 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-sysctl-d\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.458941 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458850    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e2236f3-3e04-44bd-acc0-6cda49391af4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.458941 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-hostroot\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.458941 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-run-openvswitch\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-run\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-os-release\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.458995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-etc-kubernetes\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29111c58-63e9-4233-a4ef-0983db2a6906-tmp-dir\") pod \"node-resolver-hbgp6\" (UID: \"29111c58-63e9-4233-a4ef-0983db2a6906\") " pod="openshift-dns/node-resolver-hbgp6"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459065    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-var-lib-kubelet\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfk6z\" (UniqueName: \"kubernetes.io/projected/58991d17-f442-4c6a-8d45-2c25b583bcd0-kube-api-access-pfk6z\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-var-lib-cni-bin\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-etc-openvswitch\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.459304    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-cnibin\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.459398 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.459396    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs podName:67fc1b7f-80b5-41d9-9141-9bcea29579d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:15.959371661 +0000 UTC m=+1.972751544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs") pod "network-metrics-daemon-bsr6p" (UID: "67fc1b7f-80b5-41d9-9141-9bcea29579d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-run-ovn\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e2236f3-3e04-44bd-acc0-6cda49391af4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e2236f3-3e04-44bd-acc0-6cda49391af4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vnq\" (UniqueName: \"kubernetes.io/projected/9e2236f3-3e04-44bd-acc0-6cda49391af4-kube-api-access-t8vnq\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-cni-dir\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-conf-dir\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-daemon-config\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-var-lib-openvswitch\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-cni-netd\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-lib-modules\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-registration-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-kubernetes\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-registration-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.459982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.459958    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-var-lib-cni-multus\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460007    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-slash\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-log-socket\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-os-release\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-cni-binary-copy\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-run-multus-certs\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6vsg\" (UniqueName: \"kubernetes.io/projected/67fc1b7f-80b5-41d9-9141-9bcea29579d4-kube-api-access-r6vsg\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7-konnectivity-ca\") pod \"konnectivity-agent-6vmg2\" (UID: \"ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7\") " pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-ovnkube-config\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5c24f95-13e0-4739-80ae-b6b128e520b2-dbus\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-run-netns\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460347    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ch9c\" (UniqueName: \"kubernetes.io/projected/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-kube-api-access-9ch9c\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37cc1108-c664-4196-8d56-259edfb151d5-serviceca\") pod \"node-ca-sm9zr\" (UID: \"37cc1108-c664-4196-8d56-259edfb151d5\") " pod="openshift-image-registry/node-ca-sm9zr"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-sysctl-conf\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-socket-dir-parent\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.460713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d262r\" (UniqueName: \"kubernetes.io/projected/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-kube-api-access-d262r\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-systemd\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-tuned\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460627    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-cnibin\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460669    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37cc1108-c664-4196-8d56-259edfb151d5-host\") pod \"node-ca-sm9zr\" (UID: \"37cc1108-c664-4196-8d56-259edfb151d5\") " pod="openshift-image-registry/node-ca-sm9zr"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-sys-fs\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67mg\" (UniqueName: \"kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg\") pod \"network-check-target-qllgl\" (UID: \"147f1d2b-fe6a-4336-a15c-6362fc89ec8f\") " pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-run-systemd\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-cni-bin\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-sys-fs\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5c24f95-13e0-4739-80ae-b6b128e520b2-kubelet-config\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460825    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37cc1108-c664-4196-8d56-259edfb151d5-host\") pod \"node-ca-sm9zr\" (UID: \"37cc1108-c664-4196-8d56-259edfb151d5\") " pod="openshift-image-registry/node-ca-sm9zr"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-sys\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460867    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e9eaed1-4bed-4a13-848b-d1aac51cf1c7-iptables-alerter-script\") pod \"iptables-alerter-pb79v\" (UID: \"6e9eaed1-4bed-4a13-848b-d1aac51cf1c7\") " pod="openshift-network-operator/iptables-alerter-pb79v"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-socket-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-etc-selinux\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.461599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.460990    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-ovnkube-script-lib\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58991d17-f442-4c6a-8d45-2c25b583bcd0-tmp\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461067    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-socket-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e9eaed1-4bed-4a13-848b-d1aac51cf1c7-host-slash\") pod \"iptables-alerter-pb79v\" (UID: \"6e9eaed1-4bed-4a13-848b-d1aac51cf1c7\") " pod="openshift-network-operator/iptables-alerter-pb79v"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461121    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-var-lib-kubelet\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-etc-selinux\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29111c58-63e9-4233-a4ef-0983db2a6906-hosts-file\") pod \"node-resolver-hbgp6\" (UID: \"29111c58-63e9-4233-a4ef-0983db2a6906\") " pod="openshift-dns/node-resolver-hbgp6"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-ovn-node-metrics-cert\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-host\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29111c58-63e9-4233-a4ef-0983db2a6906-hosts-file\") pod \"node-resolver-hbgp6\" (UID: \"29111c58-63e9-4233-a4ef-0983db2a6906\") " pod="openshift-dns/node-resolver-hbgp6"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8kdf\" (UniqueName: \"kubernetes.io/projected/6e9eaed1-4bed-4a13-848b-d1aac51cf1c7-kube-api-access-v8kdf\") pod \"iptables-alerter-pb79v\" (UID: \"6e9eaed1-4bed-4a13-848b-d1aac51cf1c7\") " pod="openshift-network-operator/iptables-alerter-pb79v"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-system-cni-dir\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-run-k8s-cni-cncf-io\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-device-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drgr7\" (UniqueName: \"kubernetes.io/projected/ae56b753-916d-4422-a983-1f87ffc43ec6-kube-api-access-drgr7\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xnwn\" (UniqueName: \"kubernetes.io/projected/37cc1108-c664-4196-8d56-259edfb151d5-kube-api-access-9xnwn\") pod \"node-ca-sm9zr\" (UID: \"37cc1108-c664-4196-8d56-259edfb151d5\") " pod="openshift-image-registry/node-ca-sm9zr"
Jul 28 17:17:15.462205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461450    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-kubelet\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.462717 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae56b753-916d-4422-a983-1f87ffc43ec6-device-dir\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.462717 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461500    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29111c58-63e9-4233-a4ef-0983db2a6906-tmp-dir\") pod \"node-resolver-hbgp6\" (UID: \"29111c58-63e9-4233-a4ef-0983db2a6906\") " pod="openshift-dns/node-resolver-hbgp6"
Jul 28 17:17:15.462717 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461833    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:17:15.462717 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.461967    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37cc1108-c664-4196-8d56-259edfb151d5-serviceca\") pod \"node-ca-sm9zr\" (UID: \"37cc1108-c664-4196-8d56-259edfb151d5\") " pod="openshift-image-registry/node-ca-sm9zr"
Jul 28 17:17:15.462717 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.462206    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7-konnectivity-ca\") pod \"konnectivity-agent-6vmg2\" (UID: \"ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7\") " pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:15.466149 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.466131    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7-agent-certs\") pod \"konnectivity-agent-6vmg2\" (UID: \"ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7\") " pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:15.467505 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.467473    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:15.467505 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.467498    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:15.467672 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.467513    2583 projected.go:196] Error preparing data for projected volume kube-api-access-p67mg for pod openshift-network-diagnostics/network-check-target-qllgl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:15.467672 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.467589    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg podName:147f1d2b-fe6a-4336-a15c-6362fc89ec8f nodeName:}" failed. No retries permitted until 2026-07-28 17:17:15.967569952 +0000 UTC m=+1.980949842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p67mg" (UniqueName: "kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg") pod "network-check-target-qllgl" (UID: "147f1d2b-fe6a-4336-a15c-6362fc89ec8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:15.468681 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.468653    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx6z4\" (UniqueName: \"kubernetes.io/projected/29111c58-63e9-4233-a4ef-0983db2a6906-kube-api-access-fx6z4\") pod \"node-resolver-hbgp6\" (UID: \"29111c58-63e9-4233-a4ef-0983db2a6906\") " pod="openshift-dns/node-resolver-hbgp6"
Jul 28 17:17:15.469702 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.469680    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xnwn\" (UniqueName: \"kubernetes.io/projected/37cc1108-c664-4196-8d56-259edfb151d5-kube-api-access-9xnwn\") pod \"node-ca-sm9zr\" (UID: \"37cc1108-c664-4196-8d56-259edfb151d5\") " pod="openshift-image-registry/node-ca-sm9zr"
Jul 28 17:17:15.469834 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.469815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6vsg\" (UniqueName: \"kubernetes.io/projected/67fc1b7f-80b5-41d9-9141-9bcea29579d4-kube-api-access-r6vsg\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:15.470022 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.470007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drgr7\" (UniqueName: \"kubernetes.io/projected/ae56b753-916d-4422-a983-1f87ffc43ec6-kube-api-access-drgr7\") pod \"aws-ebs-csi-driver-node-5d8tk\" (UID: \"ae56b753-916d-4422-a983-1f87ffc43ec6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.561879 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.561827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e9eaed1-4bed-4a13-848b-d1aac51cf1c7-host-slash\") pod \"iptables-alerter-pb79v\" (UID: \"6e9eaed1-4bed-4a13-848b-d1aac51cf1c7\") " pod="openshift-network-operator/iptables-alerter-pb79v"
Jul 28 17:17:15.561879 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.561875    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.561902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-var-lib-kubelet\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.561915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e9eaed1-4bed-4a13-848b-d1aac51cf1c7-host-slash\") pod \"iptables-alerter-pb79v\" (UID: \"6e9eaed1-4bed-4a13-848b-d1aac51cf1c7\") " pod="openshift-network-operator/iptables-alerter-pb79v"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.561928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-ovn-node-metrics-cert\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.561952    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-host\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.561976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kdf\" (UniqueName: \"kubernetes.io/projected/6e9eaed1-4bed-4a13-848b-d1aac51cf1c7-kube-api-access-v8kdf\") pod \"iptables-alerter-pb79v\" (UID: \"6e9eaed1-4bed-4a13-848b-d1aac51cf1c7\") " pod="openshift-network-operator/iptables-alerter-pb79v"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.561986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-var-lib-kubelet\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.561997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-system-cni-dir\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-run-k8s-cni-cncf-io\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-kubelet\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-run-netns\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-env-overrides\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-sysconfig\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562165    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562176    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-system-cni-dir\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-run-k8s-cni-cncf-io\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-systemd-units\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.562261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-node-log\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-systemd-units\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-modprobe-d\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562302    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-sysctl-d\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562314    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-kubelet\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e2236f3-3e04-44bd-acc0-6cda49391af4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-hostroot\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562355    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-run-netns\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-run-openvswitch\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-run\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-os-release\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-etc-kubernetes\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-var-lib-kubelet\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfk6z\" (UniqueName: \"kubernetes.io/projected/58991d17-f442-4c6a-8d45-2c25b583bcd0-kube-api-access-pfk6z\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562550    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-var-lib-cni-bin\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562588    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-etc-openvswitch\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.563142 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-cnibin\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562663    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-run-ovn\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e2236f3-3e04-44bd-acc0-6cda49391af4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e2236f3-3e04-44bd-acc0-6cda49391af4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562713    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-sysconfig\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vnq\" (UniqueName: \"kubernetes.io/projected/9e2236f3-3e04-44bd-acc0-6cda49391af4-kube-api-access-t8vnq\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-cni-dir\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-conf-dir\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-run\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562854    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-system-cni-dir\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562870    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-os-release\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-conf-dir\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-env-overrides\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-etc-kubernetes\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-host\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-daemon-config\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.562992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-var-lib-openvswitch\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-cnibin\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.564032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-cni-netd\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563055    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-var-lib-kubelet\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-lib-modules\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-kubernetes\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563153    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-var-lib-cni-multus\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563179    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-slash\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-var-lib-openvswitch\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563204    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-log-socket\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-os-release\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-hostroot\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-cni-binary-copy\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-run-openvswitch\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-run-multus-certs\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-run-multus-certs\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563370    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563384    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-ovnkube-config\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563412    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5c24f95-13e0-4739-80ae-b6b128e520b2-dbus\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-var-lib-cni-multus\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-run-netns\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-cni-netd\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ch9c\" (UniqueName: \"kubernetes.io/projected/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-kube-api-access-9ch9c\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563471    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-etc-openvswitch\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563516    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-daemon-config\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563545    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-log-socket\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-modprobe-d\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563572    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-sysctl-d\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563599    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-slash\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-sysctl-conf\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-os-release\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563717    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-lib-modules\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563806    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-cni-dir\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-sysctl-conf\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563844    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-run-ovn\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-socket-dir-parent\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.565828 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d262r\" (UniqueName: \"kubernetes.io/projected/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-kube-api-access-d262r\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563922    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5c24f95-13e0-4739-80ae-b6b128e520b2-dbus\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-systemd\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-tuned\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563976    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-multus-socket-dir-parent\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-cnibin\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564019    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-var-lib-cni-bin\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-run-systemd\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-ovnkube-config\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-cni-bin\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564116    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5c24f95-13e0-4739-80ae-b6b128e520b2-kubelet-config\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564128    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-host-cni-bin\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-cni-binary-copy\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564175    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-host-run-netns\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564179    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-run-systemd\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564207    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-systemd\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-sys\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.566471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564219    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e2236f3-3e04-44bd-acc0-6cda49391af4-cnibin\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e9eaed1-4bed-4a13-848b-d1aac51cf1c7-iptables-alerter-script\") pod \"iptables-alerter-pb79v\" (UID: \"6e9eaed1-4bed-4a13-848b-d1aac51cf1c7\") " pod="openshift-network-operator/iptables-alerter-pb79v"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564252    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-sys\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-ovnkube-script-lib\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58991d17-f442-4c6a-8d45-2c25b583bcd0-tmp\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564322    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e2236f3-3e04-44bd-acc0-6cda49391af4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e2236f3-3e04-44bd-acc0-6cda49391af4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.563453    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-system-cni-dir\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-kubernetes\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564590    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e2236f3-3e04-44bd-acc0-6cda49391af4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564665    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5c24f95-13e0-4739-80ae-b6b128e520b2-kubelet-config\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.564768    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564781    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-node-log\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.564838    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret podName:b5c24f95-13e0-4739-80ae-b6b128e520b2 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:16.064819952 +0000 UTC m=+2.078199846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret") pod "global-pull-secret-syncer-56htm" (UID: "b5c24f95-13e0-4739-80ae-b6b128e520b2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e9eaed1-4bed-4a13-848b-d1aac51cf1c7-iptables-alerter-script\") pod \"iptables-alerter-pb79v\" (UID: \"6e9eaed1-4bed-4a13-848b-d1aac51cf1c7\") " pod="openshift-network-operator/iptables-alerter-pb79v"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.564981    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-ovnkube-script-lib\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.566010    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-ovn-node-metrics-cert\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.567013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.566416    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58991d17-f442-4c6a-8d45-2c25b583bcd0-etc-tuned\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.567594 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.566872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58991d17-f442-4c6a-8d45-2c25b583bcd0-tmp\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.581813 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.581784    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfk6z\" (UniqueName: \"kubernetes.io/projected/58991d17-f442-4c6a-8d45-2c25b583bcd0-kube-api-access-pfk6z\") pod \"tuned-lr49l\" (UID: \"58991d17-f442-4c6a-8d45-2c25b583bcd0\") " pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.584476 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.584440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vnq\" (UniqueName: \"kubernetes.io/projected/9e2236f3-3e04-44bd-acc0-6cda49391af4-kube-api-access-t8vnq\") pod \"multus-additional-cni-plugins-5kr5s\" (UID: \"9e2236f3-3e04-44bd-acc0-6cda49391af4\") " pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.584744 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.584726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d262r\" (UniqueName: \"kubernetes.io/projected/4222a2e9-ff5a-4ebe-ac26-e782b3a58690-kube-api-access-d262r\") pod \"ovnkube-node-mdz9h\" (UID: \"4222a2e9-ff5a-4ebe-ac26-e782b3a58690\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.585904 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.585882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kdf\" (UniqueName: \"kubernetes.io/projected/6e9eaed1-4bed-4a13-848b-d1aac51cf1c7-kube-api-access-v8kdf\") pod \"iptables-alerter-pb79v\" (UID: \"6e9eaed1-4bed-4a13-848b-d1aac51cf1c7\") " pod="openshift-network-operator/iptables-alerter-pb79v"
Jul 28 17:17:15.587520 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.587498    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ch9c\" (UniqueName: \"kubernetes.io/projected/68a94d99-36cb-4e50-aa5c-da7ce1b387a7-kube-api-access-9ch9c\") pod \"multus-dzwxf\" (UID: \"68a94d99-36cb-4e50-aa5c-da7ce1b387a7\") " pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.612125 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.612084    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:15.666890 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.666840    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29924a4ae605b9b2aa658a2296cc479f.slice/crio-1385aee250fed6b12bd4cdcb817b0e271e38534aebeb2958847324e8adb79410 WatchSource:0}: Error finding container 1385aee250fed6b12bd4cdcb817b0e271e38534aebeb2958847324e8adb79410: Status 404 returned error can't find the container with id 1385aee250fed6b12bd4cdcb817b0e271e38534aebeb2958847324e8adb79410
Jul 28 17:17:15.667331 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.667310    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cc528b9c87d3a6a58c64430cacff3d.slice/crio-490bbe22122c6b8174cf6619e7642b0628c798a755ebafdc838e5b9c69c93b33 WatchSource:0}: Error finding container 490bbe22122c6b8174cf6619e7642b0628c798a755ebafdc838e5b9c69c93b33: Status 404 returned error can't find the container with id 490bbe22122c6b8174cf6619e7642b0628c798a755ebafdc838e5b9c69c93b33
Jul 28 17:17:15.672055 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.671977    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:15.672656 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.672640    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:17:15.682643 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.682619    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded18bab3_8bf4_4663_8f75_1f3a6fe2daf7.slice/crio-afbf6ee0b2351ac6a914cbdb2cfdf2c4dd7fce92f515443a19fbfa67b52797e7 WatchSource:0}: Error finding container afbf6ee0b2351ac6a914cbdb2cfdf2c4dd7fce92f515443a19fbfa67b52797e7: Status 404 returned error can't find the container with id afbf6ee0b2351ac6a914cbdb2cfdf2c4dd7fce92f515443a19fbfa67b52797e7
Jul 28 17:17:15.695059 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.695020    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk"
Jul 28 17:17:15.705179 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.705148    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae56b753_916d_4422_a983_1f87ffc43ec6.slice/crio-7377578a1c0922234f52f72c8a71b186f3f9ba38ca2dcbebc8e75f2f7ee0261a WatchSource:0}: Error finding container 7377578a1c0922234f52f72c8a71b186f3f9ba38ca2dcbebc8e75f2f7ee0261a: Status 404 returned error can't find the container with id 7377578a1c0922234f52f72c8a71b186f3f9ba38ca2dcbebc8e75f2f7ee0261a
Jul 28 17:17:15.715190 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.715167    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:15.721610 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.721591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hbgp6"
Jul 28 17:17:15.725643 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.725615    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sm9zr"
Jul 28 17:17:15.731087 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.731053    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29111c58_63e9_4233_a4ef_0983db2a6906.slice/crio-74df1e32c591bea95576dd02ee0803839df34353490fbd017232ce5ef2c6dbd3 WatchSource:0}: Error finding container 74df1e32c591bea95576dd02ee0803839df34353490fbd017232ce5ef2c6dbd3: Status 404 returned error can't find the container with id 74df1e32c591bea95576dd02ee0803839df34353490fbd017232ce5ef2c6dbd3
Jul 28 17:17:15.736499 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.736470    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cc1108_c664_4196_8d56_259edfb151d5.slice/crio-2ae42aa2bd35322edcda57a824c09de501dae037396a87b3f4cca5b65d16d833 WatchSource:0}: Error finding container 2ae42aa2bd35322edcda57a824c09de501dae037396a87b3f4cca5b65d16d833: Status 404 returned error can't find the container with id 2ae42aa2bd35322edcda57a824c09de501dae037396a87b3f4cca5b65d16d833
Jul 28 17:17:15.762811 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.762777    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5kr5s"
Jul 28 17:17:15.773008 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.772983    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2236f3_3e04_44bd_acc0_6cda49391af4.slice/crio-5983e6ea8d0e99e5ab7f1189bc0fbeb48051d09bc8b7f2459bae193e605c8c2b WatchSource:0}: Error finding container 5983e6ea8d0e99e5ab7f1189bc0fbeb48051d09bc8b7f2459bae193e605c8c2b: Status 404 returned error can't find the container with id 5983e6ea8d0e99e5ab7f1189bc0fbeb48051d09bc8b7f2459bae193e605c8c2b
Jul 28 17:17:15.789856 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.789821    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzwxf"
Jul 28 17:17:15.798924 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.798894    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a94d99_36cb_4e50_aa5c_da7ce1b387a7.slice/crio-1cff2944fff97efdb207eb72cf56319c34bd6eae5bba27d7e62522203d5f4920 WatchSource:0}: Error finding container 1cff2944fff97efdb207eb72cf56319c34bd6eae5bba27d7e62522203d5f4920: Status 404 returned error can't find the container with id 1cff2944fff97efdb207eb72cf56319c34bd6eae5bba27d7e62522203d5f4920
Jul 28 17:17:15.807673 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.807654    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pb79v"
Jul 28 17:17:15.813999 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.813982    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:15.816260 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.816235    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9eaed1_4bed_4a13_848b_d1aac51cf1c7.slice/crio-f7a22d8e6e40215b8d1ae537bb1a6d778cdc26671e50710e92929f181b5c1204 WatchSource:0}: Error finding container f7a22d8e6e40215b8d1ae537bb1a6d778cdc26671e50710e92929f181b5c1204: Status 404 returned error can't find the container with id f7a22d8e6e40215b8d1ae537bb1a6d778cdc26671e50710e92929f181b5c1204
Jul 28 17:17:15.822665 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.822641    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lr49l"
Jul 28 17:17:15.824583 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.824556    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4222a2e9_ff5a_4ebe_ac26_e782b3a58690.slice/crio-9eb313898dd64c5cf1f3f993e593f644714f1e48e5ddaa0f559c842585244dc2 WatchSource:0}: Error finding container 9eb313898dd64c5cf1f3f993e593f644714f1e48e5ddaa0f559c842585244dc2: Status 404 returned error can't find the container with id 9eb313898dd64c5cf1f3f993e593f644714f1e48e5ddaa0f559c842585244dc2
Jul 28 17:17:15.831867 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:17:15.831841    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58991d17_f442_4c6a_8d45_2c25b583bcd0.slice/crio-d2d668c88e67d372a824004812ceed1341ec85b6ea9d99ba7b7445e2dea9844c WatchSource:0}: Error finding container d2d668c88e67d372a824004812ceed1341ec85b6ea9d99ba7b7445e2dea9844c: Status 404 returned error can't find the container with id d2d668c88e67d372a824004812ceed1341ec85b6ea9d99ba7b7445e2dea9844c
Jul 28 17:17:15.833701 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.833683    2583 kuberuntime_manager.go:1459] "Unhandled Error" err="container tuned start failed in pod tuned-lr49l_openshift-cluster-node-tuning-operator(58991d17-f442-4c6a-8d45-2c25b583bcd0): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 28 17:17:15.833765 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.833748    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-lr49l" podUID="58991d17-f442-4c6a-8d45-2c25b583bcd0"
Jul 28 17:17:15.967773 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.967674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:15.967773 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:15.967733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67mg\" (UniqueName: \"kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg\") pod \"network-check-target-qllgl\" (UID: \"147f1d2b-fe6a-4336-a15c-6362fc89ec8f\") " pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:15.968027 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.967853    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:15.968027 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.967943    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs podName:67fc1b7f-80b5-41d9-9141-9bcea29579d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:16.967921313 +0000 UTC m=+2.981301200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs") pod "network-metrics-daemon-bsr6p" (UID: "67fc1b7f-80b5-41d9-9141-9bcea29579d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:15.968027 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.967857    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:15.968027 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.967972    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:15.968027 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.967982    2583 projected.go:196] Error preparing data for projected volume kube-api-access-p67mg for pod openshift-network-diagnostics/network-check-target-qllgl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:15.968324 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:15.968039    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg podName:147f1d2b-fe6a-4336-a15c-6362fc89ec8f nodeName:}" failed. No retries permitted until 2026-07-28 17:17:16.968014529 +0000 UTC m=+2.981394401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p67mg" (UniqueName: "kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg") pod "network-check-target-qllgl" (UID: "147f1d2b-fe6a-4336-a15c-6362fc89ec8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:16.068498 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.068454    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:16.068689 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:16.068666    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:16.068841 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:16.068747    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret podName:b5c24f95-13e0-4739-80ae-b6b128e520b2 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:17.068727957 +0000 UTC m=+3.082107854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret") pod "global-pull-secret-syncer-56htm" (UID: "b5c24f95-13e0-4739-80ae-b6b128e520b2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:16.406959 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.406773    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:12:16 +0000 UTC" deadline="2028-03-22 23:59:23.839880183 +0000 UTC"
Jul 28 17:17:16.406959 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.406841    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14478h42m7.433043722s"
Jul 28 17:17:16.542293 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.542169    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" event={"ID":"4222a2e9-ff5a-4ebe-ac26-e782b3a58690","Type":"ContainerStarted","Data":"9eb313898dd64c5cf1f3f993e593f644714f1e48e5ddaa0f559c842585244dc2"}
Jul 28 17:17:16.548714 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.548675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pb79v" event={"ID":"6e9eaed1-4bed-4a13-848b-d1aac51cf1c7","Type":"ContainerStarted","Data":"f7a22d8e6e40215b8d1ae537bb1a6d778cdc26671e50710e92929f181b5c1204"}
Jul 28 17:17:16.580898 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.580843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzwxf" event={"ID":"68a94d99-36cb-4e50-aa5c-da7ce1b387a7","Type":"ContainerStarted","Data":"1cff2944fff97efdb207eb72cf56319c34bd6eae5bba27d7e62522203d5f4920"}
Jul 28 17:17:16.588497 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.588455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sm9zr" event={"ID":"37cc1108-c664-4196-8d56-259edfb151d5","Type":"ContainerStarted","Data":"2ae42aa2bd35322edcda57a824c09de501dae037396a87b3f4cca5b65d16d833"}
Jul 28 17:17:16.588874 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.588853    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:16.599169 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.598428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hbgp6" event={"ID":"29111c58-63e9-4233-a4ef-0983db2a6906","Type":"ContainerStarted","Data":"74df1e32c591bea95576dd02ee0803839df34353490fbd017232ce5ef2c6dbd3"}
Jul 28 17:17:16.614005 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.613970    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk" event={"ID":"ae56b753-916d-4422-a983-1f87ffc43ec6","Type":"ContainerStarted","Data":"7377578a1c0922234f52f72c8a71b186f3f9ba38ca2dcbebc8e75f2f7ee0261a"}
Jul 28 17:17:16.636492 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.636453    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6vmg2" event={"ID":"ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7","Type":"ContainerStarted","Data":"afbf6ee0b2351ac6a914cbdb2cfdf2c4dd7fce92f515443a19fbfa67b52797e7"}
Jul 28 17:17:16.648101 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.647995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-11.ec2.internal" event={"ID":"a8cc528b9c87d3a6a58c64430cacff3d","Type":"ContainerStarted","Data":"490bbe22122c6b8174cf6619e7642b0628c798a755ebafdc838e5b9c69c93b33"}
Jul 28 17:17:16.665191 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.665106    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lr49l" event={"ID":"58991d17-f442-4c6a-8d45-2c25b583bcd0","Type":"ContainerStarted","Data":"d2d668c88e67d372a824004812ceed1341ec85b6ea9d99ba7b7445e2dea9844c"}
Jul 28 17:17:16.670872 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:16.670813    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-lr49l" podUID="58991d17-f442-4c6a-8d45-2c25b583bcd0"
Jul 28 17:17:16.680088 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.680051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kr5s" event={"ID":"9e2236f3-3e04-44bd-acc0-6cda49391af4","Type":"ContainerStarted","Data":"5983e6ea8d0e99e5ab7f1189bc0fbeb48051d09bc8b7f2459bae193e605c8c2b"}
Jul 28 17:17:16.695931 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.695895    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal" event={"ID":"29924a4ae605b9b2aa658a2296cc479f","Type":"ContainerStarted","Data":"1385aee250fed6b12bd4cdcb817b0e271e38534aebeb2958847324e8adb79410"}
Jul 28 17:17:16.976740 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.976637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67mg\" (UniqueName: \"kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg\") pod \"network-check-target-qllgl\" (UID: \"147f1d2b-fe6a-4336-a15c-6362fc89ec8f\") " pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:16.976740 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:16.976725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:16.976984 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:16.976873    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:16.976984 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:16.976901    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:16.976984 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:16.976927    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:16.976984 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:16.976942    2583 projected.go:196] Error preparing data for projected volume kube-api-access-p67mg for pod openshift-network-diagnostics/network-check-target-qllgl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:16.976984 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:16.976946    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs podName:67fc1b7f-80b5-41d9-9141-9bcea29579d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:18.97692561 +0000 UTC m=+4.990305487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs") pod "network-metrics-daemon-bsr6p" (UID: "67fc1b7f-80b5-41d9-9141-9bcea29579d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:16.977297 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:16.977001    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg podName:147f1d2b-fe6a-4336-a15c-6362fc89ec8f nodeName:}" failed. No retries permitted until 2026-07-28 17:17:18.976979725 +0000 UTC m=+4.990359615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p67mg" (UniqueName: "kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg") pod "network-check-target-qllgl" (UID: "147f1d2b-fe6a-4336-a15c-6362fc89ec8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:17.078152 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:17.078112    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:17.078417 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:17.078314    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:17.078417 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:17.078384    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret podName:b5c24f95-13e0-4739-80ae-b6b128e520b2 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:19.078365452 +0000 UTC m=+5.091745338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret") pod "global-pull-secret-syncer-56htm" (UID: "b5c24f95-13e0-4739-80ae-b6b128e520b2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:17.411395 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:17.411264    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:12:16 +0000 UTC" deadline="2027-12-27 01:23:51.71630264 +0000 UTC"
Jul 28 17:17:17.411395 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:17.411309    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12392h6m34.304998451s"
Jul 28 17:17:17.532825 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:17.531939    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:17.532825 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:17.532103    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:17.532825 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:17.532515    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:17.532825 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:17.532626    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:17.532825 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:17.532695    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:17.532825 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:17.532777    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:17.651392 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:17.651081    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:19.000329 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:18.999324    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67mg\" (UniqueName: \"kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg\") pod \"network-check-target-qllgl\" (UID: \"147f1d2b-fe6a-4336-a15c-6362fc89ec8f\") " pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:19.000329 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:18.999421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:19.000329 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:18.999581    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:19.000329 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:18.999650    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs podName:67fc1b7f-80b5-41d9-9141-9bcea29579d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:22.999629674 +0000 UTC m=+9.013009563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs") pod "network-metrics-daemon-bsr6p" (UID: "67fc1b7f-80b5-41d9-9141-9bcea29579d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:19.001066 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:19.001033    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:19.001187 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:19.001080    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:19.001187 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:19.001097    2583 projected.go:196] Error preparing data for projected volume kube-api-access-p67mg for pod openshift-network-diagnostics/network-check-target-qllgl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:19.001187 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:19.001177    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg podName:147f1d2b-fe6a-4336-a15c-6362fc89ec8f nodeName:}" failed. No retries permitted until 2026-07-28 17:17:23.001157787 +0000 UTC m=+9.014537689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p67mg" (UniqueName: "kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg") pod "network-check-target-qllgl" (UID: "147f1d2b-fe6a-4336-a15c-6362fc89ec8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:19.100710 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:19.100669    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:19.100914 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:19.100846    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:19.100981 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:19.100918    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret podName:b5c24f95-13e0-4739-80ae-b6b128e520b2 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:23.100897585 +0000 UTC m=+9.114277483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret") pod "global-pull-secret-syncer-56htm" (UID: "b5c24f95-13e0-4739-80ae-b6b128e520b2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:19.532647 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:19.532507    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:19.532647 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:19.532557    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:19.532647 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:19.532644    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:19.532950 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:19.532739    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:19.532950 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:19.532519    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:19.532950 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:19.532837    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:21.532223 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:21.532181    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:21.532747 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:21.532229    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:21.532747 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:21.532247    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:21.532747 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:21.532348    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:21.532747 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:21.532534    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:21.532747 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:21.532649    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:23.040653 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:23.040241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67mg\" (UniqueName: \"kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg\") pod \"network-check-target-qllgl\" (UID: \"147f1d2b-fe6a-4336-a15c-6362fc89ec8f\") " pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:23.040653 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:23.040318    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:23.040653 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:23.040484    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:23.040653 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:23.040546    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs podName:67fc1b7f-80b5-41d9-9141-9bcea29579d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:31.040527651 +0000 UTC m=+17.053907541 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs") pod "network-metrics-daemon-bsr6p" (UID: "67fc1b7f-80b5-41d9-9141-9bcea29579d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:23.041350 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:23.040774    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:23.041350 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:23.040801    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:23.041350 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:23.040815    2583 projected.go:196] Error preparing data for projected volume kube-api-access-p67mg for pod openshift-network-diagnostics/network-check-target-qllgl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:23.041350 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:23.040871    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg podName:147f1d2b-fe6a-4336-a15c-6362fc89ec8f nodeName:}" failed. No retries permitted until 2026-07-28 17:17:31.040850811 +0000 UTC m=+17.054230705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p67mg" (UniqueName: "kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg") pod "network-check-target-qllgl" (UID: "147f1d2b-fe6a-4336-a15c-6362fc89ec8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:23.140967 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:23.140930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:23.141174 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:23.141129    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:23.141246 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:23.141197    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret podName:b5c24f95-13e0-4739-80ae-b6b128e520b2 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:31.141174805 +0000 UTC m=+17.154554697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret") pod "global-pull-secret-syncer-56htm" (UID: "b5c24f95-13e0-4739-80ae-b6b128e520b2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:23.532263 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:23.532226    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:23.532459 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:23.532363    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:23.532459 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:23.532428    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:23.532570 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:23.532529    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:23.532626 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:23.532572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:23.532674 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:23.532642    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:25.531865 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:25.531829    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:25.531865 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:25.531850    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:25.531865 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:25.531848    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:25.532462 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:25.531976    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:25.532462 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:25.532067    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:25.532462 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:25.532138    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:27.531938 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:27.531903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:27.531938 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:27.531903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:27.532460 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:27.531903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:27.532460 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:27.532072    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:27.532460 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:27.532144    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:27.532460 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:27.532188    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:29.531771 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:29.531737    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:29.531771 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:29.531754    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:29.531771 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:29.531754    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:29.532349 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:29.532047    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:29.532349 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:29.532073    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:29.532349 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:29.532158    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:31.105857 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:31.105816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:31.106349 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:31.105884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67mg\" (UniqueName: \"kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg\") pod \"network-check-target-qllgl\" (UID: \"147f1d2b-fe6a-4336-a15c-6362fc89ec8f\") " pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:31.106349 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:31.105986    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:31.106349 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:31.106042    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:31.106349 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:31.106066    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:31.106349 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:31.106079    2583 projected.go:196] Error preparing data for projected volume kube-api-access-p67mg for pod openshift-network-diagnostics/network-check-target-qllgl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:31.106349 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:31.106082    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs podName:67fc1b7f-80b5-41d9-9141-9bcea29579d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:47.106062122 +0000 UTC m=+33.119442006 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs") pod "network-metrics-daemon-bsr6p" (UID: "67fc1b7f-80b5-41d9-9141-9bcea29579d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:31.106349 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:31.106135    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg podName:147f1d2b-fe6a-4336-a15c-6362fc89ec8f nodeName:}" failed. No retries permitted until 2026-07-28 17:17:47.106118709 +0000 UTC m=+33.119498585 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p67mg" (UniqueName: "kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg") pod "network-check-target-qllgl" (UID: "147f1d2b-fe6a-4336-a15c-6362fc89ec8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:31.206956 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:31.206912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:31.207160 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:31.207089    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:31.207237 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:31.207174    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret podName:b5c24f95-13e0-4739-80ae-b6b128e520b2 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:47.207151159 +0000 UTC m=+33.220531030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret") pod "global-pull-secret-syncer-56htm" (UID: "b5c24f95-13e0-4739-80ae-b6b128e520b2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:31.531709 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:31.531664    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:31.531709 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:31.531705    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:31.532050 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:31.531705    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:31.532050 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:31.531799    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:31.532050 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:31.531879    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:31.532050 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:31.531971    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:33.531831 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:33.531606    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:33.532223 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:33.531646    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:33.532223 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:33.531941    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:33.532223 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:33.531683    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:33.532223 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:33.532008    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:33.532223 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:33.532113    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:33.739941 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:33.739605    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzwxf" event={"ID":"68a94d99-36cb-4e50-aa5c-da7ce1b387a7","Type":"ContainerStarted","Data":"7f6d90b73ea997477ef5f826a0c0596651cb4ef5d744ab27c6a3dc29ac17c534"}
Jul 28 17:17:33.764066 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:33.763756    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-11.ec2.internal" event={"ID":"a8cc528b9c87d3a6a58c64430cacff3d","Type":"ContainerStarted","Data":"be91569083cc33e8a32d72cd05657425355acc0b34803376a4d9c1d59e5a1443"}
Jul 28 17:17:33.772369 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:33.772341    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:17:33.776438 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:33.775526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" event={"ID":"4222a2e9-ff5a-4ebe-ac26-e782b3a58690","Type":"ContainerStarted","Data":"7d271cb25de1a3cccd3dc1e7160d5e65ecf7857196c5315d3be7b9198055f2c6"}
Jul 28 17:17:33.776438 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:33.775648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" event={"ID":"4222a2e9-ff5a-4ebe-ac26-e782b3a58690","Type":"ContainerStarted","Data":"834942f7e55472953e8d137b6e9db18ffbc8c26498c0fb912db8c68dc56ecf2d"}
Jul 28 17:17:33.776438 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:33.776255    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dzwxf" podStartSLOduration=0.992559785 podStartE2EDuration="18.776238003s" totalImagesPullingTime="17.783678218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:33.776132713 +0000 UTC m=+19.789512607" watchObservedRunningTime="2026-07-28 17:17:33.776238003 +0000 UTC m=+19.789617897"
Jul 28 17:17:33.798936 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:33.798693    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-11.ec2.internal" podStartSLOduration=18.798670649 podStartE2EDuration="18.798670649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:33.798666171 +0000 UTC m=+19.812046064" watchObservedRunningTime="2026-07-28 17:17:33.798670649 +0000 UTC m=+19.812050543"
Jul 28 17:17:34.779081 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.779036    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6vmg2" event={"ID":"ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7","Type":"ContainerStarted","Data":"31a83af6a4e4336b00e945a8548e8a486dc875f35a1f96c34649d811f890b71f"}
Jul 28 17:17:34.781062 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.781014    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e2236f3-3e04-44bd-acc0-6cda49391af4" containerID="a08f76f74ab68d62c25c5307c7e78579e4ca858f8ab4ba0fc86d2db150336cfa" exitCode=0
Jul 28 17:17:34.781186 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.781053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kr5s" event={"ID":"9e2236f3-3e04-44bd-acc0-6cda49391af4","Type":"ContainerDied","Data":"a08f76f74ab68d62c25c5307c7e78579e4ca858f8ab4ba0fc86d2db150336cfa"}
Jul 28 17:17:34.782739 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.782714    2583 generic.go:356] "Generic (PLEG): container finished" podID="29924a4ae605b9b2aa658a2296cc479f" containerID="ee7822aa314b71808c5fa501279d1490cb9f1925b3b9f986e8938c1f7e56077d" exitCode=0
Jul 28 17:17:34.782822 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.782804    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal" event={"ID":"29924a4ae605b9b2aa658a2296cc479f","Type":"ContainerDied","Data":"ee7822aa314b71808c5fa501279d1490cb9f1925b3b9f986e8938c1f7e56077d"}
Jul 28 17:17:34.786110 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.786094    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:17:34.786459 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.786433    2583 generic.go:356] "Generic (PLEG): container finished" podID="4222a2e9-ff5a-4ebe-ac26-e782b3a58690" containerID="7d271cb25de1a3cccd3dc1e7160d5e65ecf7857196c5315d3be7b9198055f2c6" exitCode=1
Jul 28 17:17:34.786571 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.786496    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" event={"ID":"4222a2e9-ff5a-4ebe-ac26-e782b3a58690","Type":"ContainerDied","Data":"7d271cb25de1a3cccd3dc1e7160d5e65ecf7857196c5315d3be7b9198055f2c6"}
Jul 28 17:17:34.786626 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.786580    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" event={"ID":"4222a2e9-ff5a-4ebe-ac26-e782b3a58690","Type":"ContainerStarted","Data":"15457c3e578c9f393eda24bd761932a79118f61f5311d8d7f337c61c0687c552"}
Jul 28 17:17:34.786676 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.786631    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" event={"ID":"4222a2e9-ff5a-4ebe-ac26-e782b3a58690","Type":"ContainerStarted","Data":"4d3ee7e29b4dc44ef3d44c0f784cea5fea8860fa58e9192c1fa490e348dc5266"}
Jul 28 17:17:34.786784 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.786682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" event={"ID":"4222a2e9-ff5a-4ebe-ac26-e782b3a58690","Type":"ContainerStarted","Data":"9721238eb71997aa19bdb70f8fe6a5693ce080575bbf565b3a13dadf8a26c7ba"}
Jul 28 17:17:34.786784 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.786730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" event={"ID":"4222a2e9-ff5a-4ebe-ac26-e782b3a58690","Type":"ContainerStarted","Data":"4bdf5faa857b8311ab6a35bc35d76a8a4a13c164a6dbdf668debf93e32c19ec4"}
Jul 28 17:17:34.788336 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.788309    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sm9zr" event={"ID":"37cc1108-c664-4196-8d56-259edfb151d5","Type":"ContainerStarted","Data":"1cdd1d7c64fde5b266997ef8d8d116c78961a70608766ab7d9a6f5e0acdd4ee2"}
Jul 28 17:17:34.789971 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.789951    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hbgp6" event={"ID":"29111c58-63e9-4233-a4ef-0983db2a6906","Type":"ContainerStarted","Data":"685cd800fe8164f17e56dc443c3b3f5410d7c3ec18256ec1a5528356616d9ecd"}
Jul 28 17:17:34.791582 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.791559    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk" event={"ID":"ae56b753-916d-4422-a983-1f87ffc43ec6","Type":"ContainerStarted","Data":"d1b71283bc3ce9aee8dfcb617b3085f2d832bbfeca652725194471858039921b"}
Jul 28 17:17:34.794187 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.794151    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6vmg2" podStartSLOduration=2.29873771 podStartE2EDuration="19.794139598s" totalImagesPullingTime="17.495401888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:34.794049155 +0000 UTC m=+20.807429049" watchObservedRunningTime="2026-07-28 17:17:34.794139598 +0000 UTC m=+20.807519491"
Jul 28 17:17:34.798698 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.798677    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:34.810231 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.810169    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sm9zr" podStartSLOduration=2.368631861 podStartE2EDuration="19.810149098s" totalImagesPullingTime="17.441517237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:34.809225022 +0000 UTC m=+20.822604916" watchObservedRunningTime="2026-07-28 17:17:34.810149098 +0000 UTC m=+20.823528992"
Jul 28 17:17:34.850112 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.850055    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hbgp6" podStartSLOduration=2.404969231 podStartE2EDuration="19.850035321s" totalImagesPullingTime="17.44506609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:34.849718635 +0000 UTC m=+20.863098529" watchObservedRunningTime="2026-07-28 17:17:34.850035321 +0000 UTC m=+20.863415211"
Jul 28 17:17:34.899304 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.899060    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:34.911516 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.911484    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:17:34.999930 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:34.999896    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:35.306767 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.306738    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sm9zr_37cc1108-c664-4196-8d56-259edfb151d5/node-ca/0.log"
Jul 28 17:17:35.430813 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.430664    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:17:34.911512396Z","UUID":"226ccc0b-febb-4943-a8ef-9be6da70388c","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:17:35.432724 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.432690    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:17:35.432899 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.432735    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:17:35.531790 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.531755    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:35.531957 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.531770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:35.531957 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.531770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:35.532101 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:35.531892    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:35.532101 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:35.532033    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:35.532196 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:35.532110    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:35.795904 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.795865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pb79v" event={"ID":"6e9eaed1-4bed-4a13-848b-d1aac51cf1c7","Type":"ContainerStarted","Data":"96fa2be1cf2485a8625412e6601ca36c5ed4208ab16d33d8038810734d78497e"}
Jul 28 17:17:35.798350 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.798319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk" event={"ID":"ae56b753-916d-4422-a983-1f87ffc43ec6","Type":"ContainerStarted","Data":"c461c0959ab608162dbf1eb976e1bbf54f76de4d6a87167e58064722c5f1b6a1"}
Jul 28 17:17:35.800473 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.800449    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal" event={"ID":"29924a4ae605b9b2aa658a2296cc479f","Type":"ContainerStarted","Data":"d3c2d12f30d77784f7f6d6571fdc1de0a866d2cb9e406629c51a42c7492c7169"}
Jul 28 17:17:35.811286 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.811197    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pb79v" podStartSLOduration=3.449645431 podStartE2EDuration="20.811177017s" totalImagesPullingTime="17.361531586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:35.81080041 +0000 UTC m=+21.824180304" watchObservedRunningTime="2026-07-28 17:17:35.811177017 +0000 UTC m=+21.824556911"
Jul 28 17:17:35.826547 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:35.826244    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-11.ec2.internal" podStartSLOduration=20.826223393 podStartE2EDuration="20.826223393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:35.826001564 +0000 UTC m=+21.839381472" watchObservedRunningTime="2026-07-28 17:17:35.826223393 +0000 UTC m=+21.839603287"
Jul 28 17:17:36.032263 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:36.032174    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:36.133902 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:36.133859    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:36.271224 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:36.271176    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:36.271965 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:36.271941    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:36.805333 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:36.805289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk" event={"ID":"ae56b753-916d-4422-a983-1f87ffc43ec6","Type":"ContainerStarted","Data":"b4d526a31fb0bcb73ebc822545532873b1a100068be56fc07f8baf47925917b3"}
Jul 28 17:17:36.805877 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:36.805856    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:36.806236 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:36.806212    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6vmg2"
Jul 28 17:17:36.824855 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:36.824806    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5d8tk" podStartSLOduration=2.655134109 podStartE2EDuration="21.824787981s" totalImagesPullingTime="19.169653872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:36.824007 +0000 UTC m=+22.837386897" watchObservedRunningTime="2026-07-28 17:17:36.824787981 +0000 UTC m=+22.838167881"
Jul 28 17:17:36.873645 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:36.873589    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:37.458994 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:37.458959    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:37.519301 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:37.519262    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:37.532220 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:37.532196    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:37.532377 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:37.532314    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:37.532377 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:37.532209    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:37.532377 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:37.532200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:37.532546 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:37.532400    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:37.532546 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:37.532493    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:37.733513 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:37.733479    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:37.834949 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:37.834826    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:37.936424 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:37.936393    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:38.038278 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:38.038197    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:38.136522 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:38.136487    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:38.832569 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:38.832516    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:39.234737 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:39.234696    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:39.494870 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:39.494790    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:39.531571 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:39.531542    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:39.531739 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:39.531542    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:39.531739 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:39.531667    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:39.531739 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:39.531542    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:39.531739 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:39.531725    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:39.531941 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:39.531797    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:39.625251 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:39.625218    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:39.833086 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:39.832985    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:40.485579 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:40.485539    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:40.547238 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:40.547209    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:40.734238 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:40.734202    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:40.816833 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:40.816753    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:17:40.817145 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:40.817118    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" event={"ID":"4222a2e9-ff5a-4ebe-ac26-e782b3a58690","Type":"ContainerStarted","Data":"3d217b9eb92aaf18e4da81c85073f61e43aece7b59720dcafef1b8250137db5e"}
Jul 28 17:17:40.818398 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:40.818376    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lr49l" event={"ID":"58991d17-f442-4c6a-8d45-2c25b583bcd0","Type":"ContainerStarted","Data":"68a1c2a60b2518237681e2de94f720942f78ef96803f4cb6c64156646a6ab98f"}
Jul 28 17:17:40.820209 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:40.820189    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e2236f3-3e04-44bd-acc0-6cda49391af4" containerID="34c70e3898dd8837db89d2e8214be5d4a25358aafd75040d780d482614cd3df1" exitCode=0
Jul 28 17:17:40.820292 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:40.820220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kr5s" event={"ID":"9e2236f3-3e04-44bd-acc0-6cda49391af4","Type":"ContainerDied","Data":"34c70e3898dd8837db89d2e8214be5d4a25358aafd75040d780d482614cd3df1"}
Jul 28 17:17:40.834091 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:40.834058    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:40.836968 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:40.836924    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lr49l" podStartSLOduration=1.4210721259999999 podStartE2EDuration="25.836907648s" totalImagesPullingTime="24.415835522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-28 17:17:40.836879022 +0000 UTC m=+26.850258917" watchObservedRunningTime="2026-07-28 17:17:40.836907648 +0000 UTC m=+26.850287541"
Jul 28 17:17:41.112443 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:41.112362    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:41.232476 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:41.232442    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:41.426740 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:41.426655    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:41.528160 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:41.528121    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:41.531997 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:41.531920    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:41.532155 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:41.532047    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:41.532155 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:41.532062    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:41.532155 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:41.532079    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:41.532307 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:41.532215    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:41.532307 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:41.532286    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:41.629957 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:41.629919    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:41.732247 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:41.732213    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:41.939889 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:41.939851    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:42.826472 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.826175    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e2236f3-3e04-44bd-acc0-6cda49391af4" containerID="58f5078dba87c6d41a6d05c56e19caef8ae8ec6c013223cd046b2f8014727854" exitCode=0
Jul 28 17:17:42.827194 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.826253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kr5s" event={"ID":"9e2236f3-3e04-44bd-acc0-6cda49391af4","Type":"ContainerDied","Data":"58f5078dba87c6d41a6d05c56e19caef8ae8ec6c013223cd046b2f8014727854"}
Jul 28 17:17:42.830080 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.830061    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:17:42.830419 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.830401    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" event={"ID":"4222a2e9-ff5a-4ebe-ac26-e782b3a58690","Type":"ContainerStarted","Data":"dc4fdbe7842ac3488c8e3907b4930334769d3f5506646e9576865bdb416944cb"}
Jul 28 17:17:42.830697 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.830682    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:42.830796 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.830754    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:42.830868 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.830806    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:42.830868 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.830857    2583 scope.go:117] "RemoveContainer" containerID="7d271cb25de1a3cccd3dc1e7160d5e65ecf7857196c5315d3be7b9198055f2c6"
Jul 28 17:17:42.846640 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.846512    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:42.848215 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.848194    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:42.848302 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.848296    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:17:42.923869 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:42.923838    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:43.048247 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.048203    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:43.231917 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.231886    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:43.442762 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.442735    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:43.532406 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.532318    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:43.532406 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.532382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:43.532571 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.532382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:43.532571 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:43.532456    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:43.532571 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:43.532548    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:43.532669 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:43.532627    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:43.677072 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.677008    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:43.757244 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.757215    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:43.824859 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.824783    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:43.834340 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.834310    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e2236f3-3e04-44bd-acc0-6cda49391af4" containerID="3c791e79dba773b59586ea40c456b4cbf36e0fe9a2af567c1ca9815d68b6d4d1" exitCode=0
Jul 28 17:17:43.834824 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.834396    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kr5s" event={"ID":"9e2236f3-3e04-44bd-acc0-6cda49391af4","Type":"ContainerDied","Data":"3c791e79dba773b59586ea40c456b4cbf36e0fe9a2af567c1ca9815d68b6d4d1"}
Jul 28 17:17:43.838165 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.838144    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:17:43.838519 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.838499    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" event={"ID":"4222a2e9-ff5a-4ebe-ac26-e782b3a58690","Type":"ContainerStarted","Data":"8c9ad8b1e4b968515dd0ba6ded639927f8b6ab8a5b5acfc59c20010ee02bbc5e"}
Jul 28 17:17:43.899577 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:43.899527    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h" podStartSLOduration=11.228300976 podStartE2EDuration="28.899512527s" totalImagesPullingTime="17.671211551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:43.898573357 +0000 UTC m=+29.911953249" watchObservedRunningTime="2026-07-28 17:17:43.899512527 +0000 UTC m=+29.912892420"
Jul 28 17:17:44.633783 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:44.633747    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:45.056312 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:45.056279    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:45.164491 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:45.164455    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:45.257083 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:45.257030    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:45.360068 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:45.359968    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:45.457429 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:45.457378    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:45.532284 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:45.532249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:45.532284 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:45.532281    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:45.532498 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:45.532249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:45.532498 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:45.532357    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:45.532498 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:45.532441    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:45.532656 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:45.532513    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:45.629013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:45.628936    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:45.727753 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:45.727720    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:45.868853 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:45.868814    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:46.069333 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:46.069299    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:46.172328 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:46.172289    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:46.428961 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:46.428884    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:46.679511 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:46.679269    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:46.835895 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:46.835862    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:46.936769 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:46.936684    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:47.037229 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:47.037190    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:47.130853 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:47.130811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:47.131324 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:47.130895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67mg\" (UniqueName: \"kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg\") pod \"network-check-target-qllgl\" (UID: \"147f1d2b-fe6a-4336-a15c-6362fc89ec8f\") " pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:47.131324 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:47.130979    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:47.131324 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:47.131049    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:47.131324 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:47.131068    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs podName:67fc1b7f-80b5-41d9-9141-9bcea29579d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:19.13105217 +0000 UTC m=+65.144432057 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs") pod "network-metrics-daemon-bsr6p" (UID: "67fc1b7f-80b5-41d9-9141-9bcea29579d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:47.131324 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:47.131070    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:47.131324 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:47.131086    2583 projected.go:196] Error preparing data for projected volume kube-api-access-p67mg for pod openshift-network-diagnostics/network-check-target-qllgl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:47.131324 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:47.131136    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg podName:147f1d2b-fe6a-4336-a15c-6362fc89ec8f nodeName:}" failed. No retries permitted until 2026-07-28 17:18:19.131119497 +0000 UTC m=+65.144499374 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p67mg" (UniqueName: "kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg") pod "network-check-target-qllgl" (UID: "147f1d2b-fe6a-4336-a15c-6362fc89ec8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:47.137130 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:47.137095    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:47.232102 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:47.232072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:47.232277 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:47.232234    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:47.232350 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:47.232304    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret podName:b5c24f95-13e0-4739-80ae-b6b128e520b2 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:19.232288545 +0000 UTC m=+65.245668419 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret") pod "global-pull-secret-syncer-56htm" (UID: "b5c24f95-13e0-4739-80ae-b6b128e520b2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:47.238953 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:47.238926    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:47.532276 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:47.532193    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:47.532276 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:47.532233    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:47.532276 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:47.532233    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:47.532548 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:47.532317    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:47.532548 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:47.532426    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:47.532548 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:47.532513    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:47.632032 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:47.631982    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:48.033947 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:48.033907    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:48.231111 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:48.231072    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:48.648007 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:48.647985    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:48.840865 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:48.840792    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:48.852677 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:48.852646    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e2236f3-3e04-44bd-acc0-6cda49391af4" containerID="4c51fb3ed17781fd09b409406836ccfdd415f7f8259e3173e7c0d2ab614aa6ef" exitCode=0
Jul 28 17:17:48.852816 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:48.852698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kr5s" event={"ID":"9e2236f3-3e04-44bd-acc0-6cda49391af4","Type":"ContainerDied","Data":"4c51fb3ed17781fd09b409406836ccfdd415f7f8259e3173e7c0d2ab614aa6ef"}
Jul 28 17:17:49.507470 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:49.507440    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:49.532477 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:49.532448    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:49.532477 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:49.532475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:49.532674 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:49.532551    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:49.532674 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:49.532563    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:49.532674 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:49.532648    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:49.532815 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:49.532746    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:49.857999 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:49.857903    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e2236f3-3e04-44bd-acc0-6cda49391af4" containerID="6cf526f0552e71b2c73d1d7499285e70b19f033cab5dccc93bf05f7c7de92877" exitCode=0
Jul 28 17:17:49.857999 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:49.857968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kr5s" event={"ID":"9e2236f3-3e04-44bd-acc0-6cda49391af4","Type":"ContainerDied","Data":"6cf526f0552e71b2c73d1d7499285e70b19f033cab5dccc93bf05f7c7de92877"}
Jul 28 17:17:50.863420 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:50.863381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kr5s" event={"ID":"9e2236f3-3e04-44bd-acc0-6cda49391af4","Type":"ContainerStarted","Data":"42ac07d729d3d35136f5a724a714fedad3ec53a2ab7d1b4f95ad89e7106dd0d0"}
Jul 28 17:17:50.887382 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:50.887328    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5kr5s" podStartSLOduration=6.626470768 podStartE2EDuration="35.887313145s" totalImagesPullingTime="29.260842377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:50.885846089 +0000 UTC m=+36.899225982" watchObservedRunningTime="2026-07-28 17:17:50.887313145 +0000 UTC m=+36.900693069"
Jul 28 17:17:51.532176 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:51.532143    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:51.532350 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:51.532144    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:51.532350 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:51.532249    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:51.532350 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:51.532309    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:51.532350 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:51.532143    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:51.532501 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:51.532377    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:52.317269 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:52.317231    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:52.402595 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:52.402560    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:52.528653 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:52.528616    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:52.616218 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:52.616144    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:53.531687 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:53.531648    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:53.532157 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:53.531648    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:53.532157 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:53.531753    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:53.532157 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:53.531648    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:53.532157 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:53.531827    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:53.532157 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:53.531937    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:54.109872 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:54.109823    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:55.531533 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:55.531495    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:55.532007 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:55.531537    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:55.532007 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:55.531601    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:55.532007 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:55.531620    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:55.532007 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:55.531669    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:55.532007 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:55.531730    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:56.910333 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:56.910286    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:56.976373 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:56.976333    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:57.531927 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:57.531898    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:57.532084 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:57.531902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:57.532084 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:57.531992    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:57.532170 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:57.532080    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:17:57.532170 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:57.531902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:57.532170 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:57.532155    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:57.571518 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:57.571487    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:57.656267 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:57.656237    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:57.750271 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:57.750227    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:57.927000 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:57.926921    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:58.086277 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:58.086244    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:58.290097 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:58.290051    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:58.735160 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:58.735132    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:58.827189 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:58.827142    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:59.114505 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:59.114470    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-56htm"]
Jul 28 17:17:59.114916 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:59.114592    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:17:59.114916 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:59.114680    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:17:59.117186 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:59.117096    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bsr6p"]
Jul 28 17:17:59.117309 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:59.117201    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:17:59.117383 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:59.117305    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:17:59.118342 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:59.118323    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qllgl"]
Jul 28 17:17:59.118460 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:17:59.118442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:17:59.118566 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:17:59.118532    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:18:00.532412 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:00.532367    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:18:00.532412 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:00.532406    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:18:00.532947 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:18:00.532498    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56htm" podUID="b5c24f95-13e0-4739-80ae-b6b128e520b2"
Jul 28 17:18:00.532947 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:18:00.532603    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qllgl" podUID="147f1d2b-fe6a-4336-a15c-6362fc89ec8f"
Jul 28 17:18:01.391326 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:01.391144    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:01.489422 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:01.489390    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:01.532278 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:01.532235    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:18:01.532383 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:18:01.532364    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsr6p" podUID="67fc1b7f-80b5-41d9-9141-9bcea29579d4"
Jul 28 17:18:01.917587 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:01.917542    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:01.987070 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:01.987014    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:02.179975 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.179902    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-11.ec2.internal" event="NodeReady"
Jul 28 17:18:02.180140 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.180034    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:18:02.215009 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.214973    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b8ffcbb54-s67n4"]
Jul 28 17:18:02.219757 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.219740    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.222309 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.222291    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:18:02.222507 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.222494    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:18:02.222552 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.222523    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hn5j9\""
Jul 28 17:18:02.222552 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.222538    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:18:02.227410 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.227392    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:18:02.229967 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.229945    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b8ffcbb54-s67n4"]
Jul 28 17:18:02.232845 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.232823    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ngl5f"]
Jul 28 17:18:02.237653 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.237634    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-slzgv"]
Jul 28 17:18:02.237777 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.237762    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.240521 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.240486    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:18:02.240850 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.240569    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-842mx\""
Jul 28 17:18:02.241091 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.240619    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:18:02.247843 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.247800    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ngl5f"]
Jul 28 17:18:02.247994 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.247976    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.248883 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.248859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcf12d19-b0be-42d4-923d-b479a41100f3-ca-trust-extracted\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.249120 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcf12d19-b0be-42d4-923d-b479a41100f3-bound-sa-token\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.249258 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249237    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-slzgv"]
Jul 28 17:18:02.249364 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcf12d19-b0be-42d4-923d-b479a41100f3-image-registry-private-configuration\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.249503 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcf12d19-b0be-42d4-923d-b479a41100f3-trusted-ca\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.249503 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcf12d19-b0be-42d4-923d-b479a41100f3-installation-pull-secrets\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.249503 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3fb048-2464-4ef1-8f96-626c3154af68-tmp-dir\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.249503 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcf12d19-b0be-42d4-923d-b479a41100f3-registry-certificates\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.249714 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249509    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f3fb048-2464-4ef1-8f96-626c3154af68-metrics-tls\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.249714 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tv8d\" (UniqueName: \"kubernetes.io/projected/3f3fb048-2464-4ef1-8f96-626c3154af68-kube-api-access-6tv8d\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.249714 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249568    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdkcp\" (UniqueName: \"kubernetes.io/projected/bcf12d19-b0be-42d4-923d-b479a41100f3-kube-api-access-mdkcp\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.249714 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3fb048-2464-4ef1-8f96-626c3154af68-config-volume\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.249714 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.249664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf12d19-b0be-42d4-923d-b479a41100f3-registry-tls\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.250681 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.250665    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:18:02.250811 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.250667    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kdvg7\""
Jul 28 17:18:02.251423 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.251403    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:18:02.251716 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.251683    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:18:02.251821 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.251803    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:18:02.331467 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.331432    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lq8bf"]
Jul 28 17:18:02.344341 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.344311    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lq8bf"]
Jul 28 17:18:02.344496 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.344403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lq8bf"
Jul 28 17:18:02.347012 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.346986    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:18:02.347230 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.347044    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rk7tv\""
Jul 28 17:18:02.347230 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.347090    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:18:02.347230 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.347121    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:18:02.350238 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcf12d19-b0be-42d4-923d-b479a41100f3-ca-trust-extracted\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.350320 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcf12d19-b0be-42d4-923d-b479a41100f3-bound-sa-token\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.350320 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350280    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd0d8257-ed38-4f9f-b067-c4302cac2b24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.350412 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350395    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lwgk\" (UniqueName: \"kubernetes.io/projected/cd0d8257-ed38-4f9f-b067-c4302cac2b24-kube-api-access-7lwgk\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.350457 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd0d8257-ed38-4f9f-b067-c4302cac2b24-crio-socket\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.350501 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd0d8257-ed38-4f9f-b067-c4302cac2b24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.350556 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcf12d19-b0be-42d4-923d-b479a41100f3-image-registry-private-configuration\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.350611 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcf12d19-b0be-42d4-923d-b479a41100f3-trusted-ca\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.350611 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcf12d19-b0be-42d4-923d-b479a41100f3-installation-pull-secrets\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.350611 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350599    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcf12d19-b0be-42d4-923d-b479a41100f3-ca-trust-extracted\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.350955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kt6s\" (UniqueName: \"kubernetes.io/projected/308f491b-d382-4e27-9bf5-36b72d56d07b-kube-api-access-2kt6s\") pod \"ingress-canary-lq8bf\" (UID: \"308f491b-d382-4e27-9bf5-36b72d56d07b\") " pod="openshift-ingress-canary/ingress-canary-lq8bf"
Jul 28 17:18:02.350955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3fb048-2464-4ef1-8f96-626c3154af68-tmp-dir\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.350955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350683    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcf12d19-b0be-42d4-923d-b479a41100f3-registry-certificates\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.350955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350720    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f3fb048-2464-4ef1-8f96-626c3154af68-metrics-tls\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.350955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tv8d\" (UniqueName: \"kubernetes.io/projected/3f3fb048-2464-4ef1-8f96-626c3154af68-kube-api-access-6tv8d\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.350955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350770    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdkcp\" (UniqueName: \"kubernetes.io/projected/bcf12d19-b0be-42d4-923d-b479a41100f3-kube-api-access-mdkcp\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.350955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3fb048-2464-4ef1-8f96-626c3154af68-config-volume\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.350955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350832    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd0d8257-ed38-4f9f-b067-c4302cac2b24-data-volume\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.350955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/308f491b-d382-4e27-9bf5-36b72d56d07b-cert\") pod \"ingress-canary-lq8bf\" (UID: \"308f491b-d382-4e27-9bf5-36b72d56d07b\") " pod="openshift-ingress-canary/ingress-canary-lq8bf"
Jul 28 17:18:02.350955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.350882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf12d19-b0be-42d4-923d-b479a41100f3-registry-tls\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.351436 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.351118    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3fb048-2464-4ef1-8f96-626c3154af68-tmp-dir\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.351487 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.351437    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcf12d19-b0be-42d4-923d-b479a41100f3-trusted-ca\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.352083 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.352062    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3fb048-2464-4ef1-8f96-626c3154af68-config-volume\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.352336 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.352312    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcf12d19-b0be-42d4-923d-b479a41100f3-registry-certificates\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.354958 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.354933    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcf12d19-b0be-42d4-923d-b479a41100f3-image-registry-private-configuration\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.354958 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.354955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f3fb048-2464-4ef1-8f96-626c3154af68-metrics-tls\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.355119 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.354933    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcf12d19-b0be-42d4-923d-b479a41100f3-installation-pull-secrets\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.355119 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.355048    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf12d19-b0be-42d4-923d-b479a41100f3-registry-tls\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.358643 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.358613    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcf12d19-b0be-42d4-923d-b479a41100f3-bound-sa-token\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.358724 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.358673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdkcp\" (UniqueName: \"kubernetes.io/projected/bcf12d19-b0be-42d4-923d-b479a41100f3-kube-api-access-mdkcp\") pod \"image-registry-5b8ffcbb54-s67n4\" (UID: \"bcf12d19-b0be-42d4-923d-b479a41100f3\") " pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.359033 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.359003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tv8d\" (UniqueName: \"kubernetes.io/projected/3f3fb048-2464-4ef1-8f96-626c3154af68-kube-api-access-6tv8d\") pod \"dns-default-ngl5f\" (UID: \"3f3fb048-2464-4ef1-8f96-626c3154af68\") " pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.362794 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.362770    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:02.425982 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.425947    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:02.451416 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.451322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd0d8257-ed38-4f9f-b067-c4302cac2b24-data-volume\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.451416 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.451362    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/308f491b-d382-4e27-9bf5-36b72d56d07b-cert\") pod \"ingress-canary-lq8bf\" (UID: \"308f491b-d382-4e27-9bf5-36b72d56d07b\") " pod="openshift-ingress-canary/ingress-canary-lq8bf"
Jul 28 17:18:02.451638 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.451421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd0d8257-ed38-4f9f-b067-c4302cac2b24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.451638 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.451444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lwgk\" (UniqueName: \"kubernetes.io/projected/cd0d8257-ed38-4f9f-b067-c4302cac2b24-kube-api-access-7lwgk\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.451638 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.451469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd0d8257-ed38-4f9f-b067-c4302cac2b24-crio-socket\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.451638 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.451491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd0d8257-ed38-4f9f-b067-c4302cac2b24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.451638 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.451527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kt6s\" (UniqueName: \"kubernetes.io/projected/308f491b-d382-4e27-9bf5-36b72d56d07b-kube-api-access-2kt6s\") pod \"ingress-canary-lq8bf\" (UID: \"308f491b-d382-4e27-9bf5-36b72d56d07b\") " pod="openshift-ingress-canary/ingress-canary-lq8bf"
Jul 28 17:18:02.451834 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.451745    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd0d8257-ed38-4f9f-b067-c4302cac2b24-crio-socket\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.451834 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.451754    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd0d8257-ed38-4f9f-b067-c4302cac2b24-data-volume\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.452079 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.452050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd0d8257-ed38-4f9f-b067-c4302cac2b24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.453815 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.453794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd0d8257-ed38-4f9f-b067-c4302cac2b24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.453908 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.453846    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/308f491b-d382-4e27-9bf5-36b72d56d07b-cert\") pod \"ingress-canary-lq8bf\" (UID: \"308f491b-d382-4e27-9bf5-36b72d56d07b\") " pod="openshift-ingress-canary/ingress-canary-lq8bf"
Jul 28 17:18:02.466146 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.466121    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lwgk\" (UniqueName: \"kubernetes.io/projected/cd0d8257-ed38-4f9f-b067-c4302cac2b24-kube-api-access-7lwgk\") pod \"insights-runtime-extractor-slzgv\" (UID: \"cd0d8257-ed38-4f9f-b067-c4302cac2b24\") " pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.467842 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.467823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kt6s\" (UniqueName: \"kubernetes.io/projected/308f491b-d382-4e27-9bf5-36b72d56d07b-kube-api-access-2kt6s\") pod \"ingress-canary-lq8bf\" (UID: \"308f491b-d382-4e27-9bf5-36b72d56d07b\") " pod="openshift-ingress-canary/ingress-canary-lq8bf"
Jul 28 17:18:02.529737 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.529695    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.531941 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.531923    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:18:02.532154 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.532140    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:18:02.534904 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.534853    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:18:02.534904 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.534902    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:18:02.535104 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.534910    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9lx2z\""
Jul 28 17:18:02.535104 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.534944    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:18:02.547334 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.547316    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:02.557606 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.557586    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-slzgv"
Jul 28 17:18:02.654536 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.653914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lq8bf"
Jul 28 17:18:02.744176 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.744129    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:02.765822 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.765790    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b8ffcbb54-s67n4"]
Jul 28 17:18:02.770584 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:02.770547    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf12d19_b0be_42d4_923d_b479a41100f3.slice/crio-030b1146e91bedd653633199d4788cbe92b597203f8d6d8a08abba7d60f3f5dd WatchSource:0}: Error finding container 030b1146e91bedd653633199d4788cbe92b597203f8d6d8a08abba7d60f3f5dd: Status 404 returned error can't find the container with id 030b1146e91bedd653633199d4788cbe92b597203f8d6d8a08abba7d60f3f5dd
Jul 28 17:18:02.778469 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.777800    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ngl5f"]
Jul 28 17:18:02.778747 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.778572    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-slzgv"]
Jul 28 17:18:02.780587 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:02.780557    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0d8257_ed38_4f9f_b067_c4302cac2b24.slice/crio-56e2c0c5c42d3fa02ae219ba5ba25637057e5d279c49400c3e84393d0be6e526 WatchSource:0}: Error finding container 56e2c0c5c42d3fa02ae219ba5ba25637057e5d279c49400c3e84393d0be6e526: Status 404 returned error can't find the container with id 56e2c0c5c42d3fa02ae219ba5ba25637057e5d279c49400c3e84393d0be6e526
Jul 28 17:18:02.781523 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:02.781460    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3fb048_2464_4ef1_8f96_626c3154af68.slice/crio-8d06a9610433da824419aee74a96ec594c1725608ac3151a8c93d1301190eefa WatchSource:0}: Error finding container 8d06a9610433da824419aee74a96ec594c1725608ac3151a8c93d1301190eefa: Status 404 returned error can't find the container with id 8d06a9610433da824419aee74a96ec594c1725608ac3151a8c93d1301190eefa
Jul 28 17:18:02.819109 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.818819    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lq8bf"]
Jul 28 17:18:02.822176 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:02.822154    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308f491b_d382_4e27_9bf5_36b72d56d07b.slice/crio-ead81f9d5eb195598ccb9ede709409867d28c38ab2edc6d191732ed116ca1970 WatchSource:0}: Error finding container ead81f9d5eb195598ccb9ede709409867d28c38ab2edc6d191732ed116ca1970: Status 404 returned error can't find the container with id ead81f9d5eb195598ccb9ede709409867d28c38ab2edc6d191732ed116ca1970
Jul 28 17:18:02.823231 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.823201    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:02.893736 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.893697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lq8bf" event={"ID":"308f491b-d382-4e27-9bf5-36b72d56d07b","Type":"ContainerStarted","Data":"ead81f9d5eb195598ccb9ede709409867d28c38ab2edc6d191732ed116ca1970"}
Jul 28 17:18:02.894824 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.894799    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-slzgv" event={"ID":"cd0d8257-ed38-4f9f-b067-c4302cac2b24","Type":"ContainerStarted","Data":"56e2c0c5c42d3fa02ae219ba5ba25637057e5d279c49400c3e84393d0be6e526"}
Jul 28 17:18:02.895846 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.895821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ngl5f" event={"ID":"3f3fb048-2464-4ef1-8f96-626c3154af68","Type":"ContainerStarted","Data":"8d06a9610433da824419aee74a96ec594c1725608ac3151a8c93d1301190eefa"}
Jul 28 17:18:02.897193 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.897174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4" event={"ID":"bcf12d19-b0be-42d4-923d-b479a41100f3","Type":"ContainerStarted","Data":"657583ecd2aac3f0597400c34bcfbaba94b8e211fa431351cbccfb1f4494e6a1"}
Jul 28 17:18:02.897273 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.897207    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4" event={"ID":"bcf12d19-b0be-42d4-923d-b479a41100f3","Type":"ContainerStarted","Data":"030b1146e91bedd653633199d4788cbe92b597203f8d6d8a08abba7d60f3f5dd"}
Jul 28 17:18:02.897335 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.897314    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:02.917581 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:02.917524    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4" podStartSLOduration=5.9175086199999996 podStartE2EDuration="5.91750862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:02.916807522 +0000 UTC m=+48.930187416" watchObservedRunningTime="2026-07-28 17:18:02.91750862 +0000 UTC m=+48.930888540"
Jul 28 17:18:03.231294 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:03.231211    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:03.331075 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:03.330993    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:03.407871 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:03.407789    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:03.531624 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:03.531528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:18:03.533186 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:03.533156    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:03.535440 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:03.535418    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmkkk\""
Jul 28 17:18:03.535709 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:03.535691    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:18:04.521315 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:04.521271    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:04.588322 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:04.588282    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:07.017820 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:07.017782    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:07.088223 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:07.088164    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:08.914449 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:08.914274    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lq8bf" event={"ID":"308f491b-d382-4e27-9bf5-36b72d56d07b","Type":"ContainerStarted","Data":"a2b871cfbc13d491cb06fc7527b16db7e62ca1c833dfe480e8610efa70d1d4e1"}
Jul 28 17:18:08.915579 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:08.915560    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ngl5f" event={"ID":"3f3fb048-2464-4ef1-8f96-626c3154af68","Type":"ContainerStarted","Data":"61f11d82da81f640969b2773bc6439ff0e7e8c586ffc8786ab20b42746139f6d"}
Jul 28 17:18:08.932135 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:08.932088    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lq8bf" podStartSLOduration=1.140288186 podStartE2EDuration="6.932068599s" totalImagesPullingTime="5.791780413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:08.930172573 +0000 UTC m=+54.943552505" watchObservedRunningTime="2026-07-28 17:18:08.932068599 +0000 UTC m=+54.945448503"
Jul 28 17:18:09.921698 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:09.921660    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ngl5f" event={"ID":"3f3fb048-2464-4ef1-8f96-626c3154af68","Type":"ContainerStarted","Data":"09a62cf4e92b15bdf696d8d337000f95482c6a738b95b9a55b483f1b7edca982"}
Jul 28 17:18:09.922396 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:09.921778    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:09.923342 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:09.923317    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-slzgv" event={"ID":"cd0d8257-ed38-4f9f-b067-c4302cac2b24","Type":"ContainerStarted","Data":"9da38531f3a33347638033fd16db56e238b0d1560543e295b0eac18e54e97931"}
Jul 28 17:18:09.940877 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:09.940833    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ngl5f" podStartSLOduration=2.108570449 podStartE2EDuration="7.940815256s" totalImagesPullingTime="5.832244807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:09.940303672 +0000 UTC m=+55.953683567" watchObservedRunningTime="2026-07-28 17:18:09.940815256 +0000 UTC m=+55.954195127"
Jul 28 17:18:12.017419 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.017385    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:12.121713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.121668    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:12.200096 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.200065    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz"]
Jul 28 17:18:12.209227 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.209204    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz"
Jul 28 17:18:12.212895 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.212864    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 17:18:12.213053 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.212987    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bmt7t\""
Jul 28 17:18:12.215512 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.215489    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz"]
Jul 28 17:18:12.224123 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.224099    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:12.336551 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.336465    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/888fc068-1889-45eb-b4ae-bc1920e4a9ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-wz4jz\" (UID: \"888fc068-1889-45eb-b4ae-bc1920e4a9ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz"
Jul 28 17:18:12.437836 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.437784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/888fc068-1889-45eb-b4ae-bc1920e4a9ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-wz4jz\" (UID: \"888fc068-1889-45eb-b4ae-bc1920e4a9ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz"
Jul 28 17:18:12.438011 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:18:12.437957    2583 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 17:18:12.438102 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:18:12.438090    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/888fc068-1889-45eb-b4ae-bc1920e4a9ae-tls-certificates podName:888fc068-1889-45eb-b4ae-bc1920e4a9ae nodeName:}" failed. No retries permitted until 2026-07-28 17:18:12.938012606 +0000 UTC m=+58.951392480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/888fc068-1889-45eb-b4ae-bc1920e4a9ae-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-wz4jz" (UID: "888fc068-1889-45eb-b4ae-bc1920e4a9ae") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 17:18:12.571264 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.571231    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:12.671666 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.671583    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:12.941469 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.941451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/888fc068-1889-45eb-b4ae-bc1920e4a9ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-wz4jz\" (UID: \"888fc068-1889-45eb-b4ae-bc1920e4a9ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz"
Jul 28 17:18:12.946109 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:12.946083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/888fc068-1889-45eb-b4ae-bc1920e4a9ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-wz4jz\" (UID: \"888fc068-1889-45eb-b4ae-bc1920e4a9ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz"
Jul 28 17:18:13.119879 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:13.119841    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz"
Jul 28 17:18:13.263735 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:13.263563    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888fc068_1889_45eb_b4ae_bc1920e4a9ae.slice/crio-9497e35bfe0eeb9aaf6b5d88fc99324c102288ddf175297b79248f41d003450b WatchSource:0}: Error finding container 9497e35bfe0eeb9aaf6b5d88fc99324c102288ddf175297b79248f41d003450b: Status 404 returned error can't find the container with id 9497e35bfe0eeb9aaf6b5d88fc99324c102288ddf175297b79248f41d003450b
Jul 28 17:18:13.263735 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:13.263708    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz"]
Jul 28 17:18:13.550342 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:13.550304    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:13.634382 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:13.634348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:13.939527 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:13.939430    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-slzgv" event={"ID":"cd0d8257-ed38-4f9f-b067-c4302cac2b24","Type":"ContainerStarted","Data":"d3878dbe9e0e9919f8f322d65342ec4384eee2c89c30b760948562a217fd1ef4"}
Jul 28 17:18:13.940880 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:13.940848    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz" event={"ID":"888fc068-1889-45eb-b4ae-bc1920e4a9ae","Type":"ContainerStarted","Data":"9497e35bfe0eeb9aaf6b5d88fc99324c102288ddf175297b79248f41d003450b"}
Jul 28 17:18:14.631392 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:14.631300    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:14.694038 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:14.693983    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:14.853515 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:14.853488    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdz9h"
Jul 28 17:18:14.946415 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:14.946334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz" event={"ID":"888fc068-1889-45eb-b4ae-bc1920e4a9ae","Type":"ContainerStarted","Data":"770959777ec931525b9916b2cd7c9e8d9f7d0c2b87afa1d387783645938fd72f"}
Jul 28 17:18:14.946584 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:14.946527    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz"
Jul 28 17:18:14.952776 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:14.952747    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz"
Jul 28 17:18:14.963624 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:14.963580    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wz4jz" podStartSLOduration=1.931860423 podStartE2EDuration="2.963562456s" totalImagesPullingTime="1.031702033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:14.962279571 +0000 UTC m=+60.975659464" watchObservedRunningTime="2026-07-28 17:18:14.963562456 +0000 UTC m=+60.976942358"
Jul 28 17:18:15.013840 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:15.013793    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:15.072884 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:15.072850    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:15.141343 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:15.141306    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:15.243778 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:15.243743    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:15.571295 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:15.571210    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:16.118238 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:16.118197    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:16.954940 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:16.954839    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-slzgv" event={"ID":"cd0d8257-ed38-4f9f-b067-c4302cac2b24","Type":"ContainerStarted","Data":"d259ea295f1f0718ad998540ea28bd6ece695450b0ab7b5ed404ac73406ab6d6"}
Jul 28 17:18:16.963394 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:16.963342    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:16.973782 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:16.973737    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-slzgv" podStartSLOduration=1.389015986 podStartE2EDuration="14.973719144s" totalImagesPullingTime="13.584703158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:18:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:16.972766218 +0000 UTC m=+62.986146108" watchObservedRunningTime="2026-07-28 17:18:16.973719144 +0000 UTC m=+62.987099037"
Jul 28 17:18:17.065218 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:17.065180    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:19.192963 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.192924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:18:19.193387 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.192990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67mg\" (UniqueName: \"kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg\") pod \"network-check-target-qllgl\" (UID: \"147f1d2b-fe6a-4336-a15c-6362fc89ec8f\") " pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:18:19.195479 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.195462    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:18:19.195533 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.195506    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:18:19.205394 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.205371    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:18:19.206551 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.206531    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fc1b7f-80b5-41d9-9141-9bcea29579d4-metrics-certs\") pod \"network-metrics-daemon-bsr6p\" (UID: \"67fc1b7f-80b5-41d9-9141-9bcea29579d4\") " pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:18:19.215429 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.215406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67mg\" (UniqueName: \"kubernetes.io/projected/147f1d2b-fe6a-4336-a15c-6362fc89ec8f-kube-api-access-p67mg\") pod \"network-check-target-qllgl\" (UID: \"147f1d2b-fe6a-4336-a15c-6362fc89ec8f\") " pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:18:19.293981 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.293944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:18:19.296624 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.296609    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:18:19.306489 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.306461    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5c24f95-13e0-4739-80ae-b6b128e520b2-original-pull-secret\") pod \"global-pull-secret-syncer-56htm\" (UID: \"b5c24f95-13e0-4739-80ae-b6b128e520b2\") " pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:18:19.344711 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.344682    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9lx2z\""
Jul 28 17:18:19.346761 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.346732    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56htm"
Jul 28 17:18:19.352859 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.352839    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:18:19.448137 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.448108    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmkkk\""
Jul 28 17:18:19.455394 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.455365    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsr6p"
Jul 28 17:18:19.502462 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.502436    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-56htm"]
Jul 28 17:18:19.504476 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:19.504447    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c24f95_13e0_4739_80ae_b6b128e520b2.slice/crio-59e0c0c255042d8b9ec66856a28ee08680ef18e461bf4890135cdfce55522e4b WatchSource:0}: Error finding container 59e0c0c255042d8b9ec66856a28ee08680ef18e461bf4890135cdfce55522e4b: Status 404 returned error can't find the container with id 59e0c0c255042d8b9ec66856a28ee08680ef18e461bf4890135cdfce55522e4b
Jul 28 17:18:19.529879 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.529850    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qllgl"]
Jul 28 17:18:19.533869 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:19.533838    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147f1d2b_fe6a_4336_a15c_6362fc89ec8f.slice/crio-53ed628c11a7ad83c545ac222d87c1bfd37756e6125dd791de50f441f20f10a2 WatchSource:0}: Error finding container 53ed628c11a7ad83c545ac222d87c1bfd37756e6125dd791de50f441f20f10a2: Status 404 returned error can't find the container with id 53ed628c11a7ad83c545ac222d87c1bfd37756e6125dd791de50f441f20f10a2
Jul 28 17:18:19.606846 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.606814    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bsr6p"]
Jul 28 17:18:19.609928 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:19.609904    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fc1b7f_80b5_41d9_9141_9bcea29579d4.slice/crio-b9eb338fce3c2da8a4f301869e8dab82686e10caece8f56d9c44f8a331204210 WatchSource:0}: Error finding container b9eb338fce3c2da8a4f301869e8dab82686e10caece8f56d9c44f8a331204210: Status 404 returned error can't find the container with id b9eb338fce3c2da8a4f301869e8dab82686e10caece8f56d9c44f8a331204210
Jul 28 17:18:19.930164 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.930135    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ngl5f"
Jul 28 17:18:19.971650 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.971555    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bsr6p" event={"ID":"67fc1b7f-80b5-41d9-9141-9bcea29579d4","Type":"ContainerStarted","Data":"b9eb338fce3c2da8a4f301869e8dab82686e10caece8f56d9c44f8a331204210"}
Jul 28 17:18:19.973256 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.973183    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qllgl" event={"ID":"147f1d2b-fe6a-4336-a15c-6362fc89ec8f","Type":"ContainerStarted","Data":"53ed628c11a7ad83c545ac222d87c1bfd37756e6125dd791de50f441f20f10a2"}
Jul 28 17:18:19.975086 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:19.975001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-56htm" event={"ID":"b5c24f95-13e0-4739-80ae-b6b128e520b2","Type":"ContainerStarted","Data":"59e0c0c255042d8b9ec66856a28ee08680ef18e461bf4890135cdfce55522e4b"}
Jul 28 17:18:20.441664 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:20.441602    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:20.546949 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:20.546889    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:20.641987 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:20.641941    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:20.982068 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:20.981972    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bsr6p" event={"ID":"67fc1b7f-80b5-41d9-9141-9bcea29579d4","Type":"ContainerStarted","Data":"90bf367f20a1833574acf08305cd0deccd3044ecd0c95bee360b8bbec6aa0b22"}
Jul 28 17:18:20.982068 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:20.982049    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bsr6p" event={"ID":"67fc1b7f-80b5-41d9-9141-9bcea29579d4","Type":"ContainerStarted","Data":"0f9db8284250cdc79b7454ec46ba002391e1e4989473d5dcf6f652baff42a887"}
Jul 28 17:18:21.004059 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:21.003610    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bsr6p" podStartSLOduration=64.892628809 podStartE2EDuration="1m6.003588737s" totalImagesPullingTime="1.110959928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:21.001320626 +0000 UTC m=+67.014700520" watchObservedRunningTime="2026-07-28 17:18:21.003588737 +0000 UTC m=+67.016968631"
Jul 28 17:18:21.120112 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:21.120070    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:21.192417 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:21.192369    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:22.401542 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.401499    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:22.500316 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.500285    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:22.535479 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.535439    2583 patch_prober.go:28] interesting pod/image-registry-5b8ffcbb54-s67n4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 17:18:22.535659 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.535504    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4" podUID="bcf12d19-b0be-42d4-923d-b479a41100f3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:18:22.602834 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.602794    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:22.681749 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.681653    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"]
Jul 28 17:18:22.684103 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.684084    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.686735 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.686715    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:18:22.688223 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.688202    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 17:18:22.688359 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.688336    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 17:18:22.688359 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.688336    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:18:22.688582 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.688430    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-x78tj\""
Jul 28 17:18:22.688921 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.688904    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:18:22.702900 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.702873    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"]
Jul 28 17:18:22.709425 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.709315    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:22.717005 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.716970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b17bc288-c3fc-4fee-8500-f0e61dfd3729-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.717169 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.717114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b17bc288-c3fc-4fee-8500-f0e61dfd3729-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.717169 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.717161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frx47\" (UniqueName: \"kubernetes.io/projected/b17bc288-c3fc-4fee-8500-f0e61dfd3729-kube-api-access-frx47\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.717255 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.717207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b17bc288-c3fc-4fee-8500-f0e61dfd3729-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.749819 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.749789    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"]
Jul 28 17:18:22.752146 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.752122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.755244 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.755218    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 17:18:22.755650 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.755467    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 17:18:22.755650 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.755498    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5mwnj\""
Jul 28 17:18:22.755650 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.755559    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 17:18:22.762431 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.762406    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jlr74"]
Jul 28 17:18:22.764546 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.764527    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.767588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.767564    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:18:22.767810 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.767710    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pd78l\""
Jul 28 17:18:22.768222 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.768201    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:18:22.768319 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.768241    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:18:22.773401 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.773354    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"]
Jul 28 17:18:22.807111 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.807055    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:22.818063 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b17bc288-c3fc-4fee-8500-f0e61dfd3729-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.818241 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818073    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/991b8a71-6107-4f4c-8abc-a3136e76f7cb-root\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.818241 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd4ld\" (UniqueName: \"kubernetes.io/projected/9d96b046-bc87-4458-8895-ff39f8febb53-kube-api-access-fd4ld\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.818241 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b17bc288-c3fc-4fee-8500-f0e61dfd3729-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.818241 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818173    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.818241 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.818241 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-wtmp\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.818241 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9d96b046-bc87-4458-8895-ff39f8febb53-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.818588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d96b046-bc87-4458-8895-ff39f8febb53-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.818588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991b8a71-6107-4f4c-8abc-a3136e76f7cb-sys\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.818588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8929l\" (UniqueName: \"kubernetes.io/projected/991b8a71-6107-4f4c-8abc-a3136e76f7cb-kube-api-access-8929l\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.818588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.818588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frx47\" (UniqueName: \"kubernetes.io/projected/b17bc288-c3fc-4fee-8500-f0e61dfd3729-kube-api-access-frx47\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.818588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-textfile\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.818588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.818868 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.818868 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-tls\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.818868 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/991b8a71-6107-4f4c-8abc-a3136e76f7cb-metrics-client-ca\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.818868 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b17bc288-c3fc-4fee-8500-f0e61dfd3729-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.818868 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:18:22.818859    2583 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 17:18:22.819086 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:18:22.818915    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b17bc288-c3fc-4fee-8500-f0e61dfd3729-openshift-state-metrics-tls podName:b17bc288-c3fc-4fee-8500-f0e61dfd3729 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:23.318895444 +0000 UTC m=+69.332275315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b17bc288-c3fc-4fee-8500-f0e61dfd3729-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-47z74" (UID: "b17bc288-c3fc-4fee-8500-f0e61dfd3729") : secret "openshift-state-metrics-tls" not found
Jul 28 17:18:22.819086 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.818937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b17bc288-c3fc-4fee-8500-f0e61dfd3729-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.820764 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.820742    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b17bc288-c3fc-4fee-8500-f0e61dfd3729-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.833516 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.833480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frx47\" (UniqueName: \"kubernetes.io/projected/b17bc288-c3fc-4fee-8500-f0e61dfd3729-kube-api-access-frx47\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:22.919908 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.919869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-tls\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.919908 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.919919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/991b8a71-6107-4f4c-8abc-a3136e76f7cb-metrics-client-ca\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920177 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.919978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/991b8a71-6107-4f4c-8abc-a3136e76f7cb-root\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920177 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd4ld\" (UniqueName: \"kubernetes.io/projected/9d96b046-bc87-4458-8895-ff39f8febb53-kube-api-access-fd4ld\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.920177 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920177 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.920177 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920100    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/991b8a71-6107-4f4c-8abc-a3136e76f7cb-root\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920177 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-wtmp\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920177 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9d96b046-bc87-4458-8895-ff39f8febb53-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.920471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d96b046-bc87-4458-8895-ff39f8febb53-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.920471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991b8a71-6107-4f4c-8abc-a3136e76f7cb-sys\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920251    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-wtmp\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920257    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8929l\" (UniqueName: \"kubernetes.io/projected/991b8a71-6107-4f4c-8abc-a3136e76f7cb-kube-api-access-8929l\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-textfile\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.920471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.920847 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:18:22.920495    2583 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 28 17:18:22.920847 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:18:22.920547    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-tls podName:9d96b046-bc87-4458-8895-ff39f8febb53 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:23.420530287 +0000 UTC m=+69.433910159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-fcthp" (UID: "9d96b046-bc87-4458-8895-ff39f8febb53") : secret "kube-state-metrics-tls" not found
Jul 28 17:18:22.920847 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991b8a71-6107-4f4c-8abc-a3136e76f7cb-sys\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920847 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/991b8a71-6107-4f4c-8abc-a3136e76f7cb-metrics-client-ca\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.920847 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920687    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-textfile\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.921111 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.920875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9d96b046-bc87-4458-8895-ff39f8febb53-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.921169 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.921137    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.921224 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.921163    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d96b046-bc87-4458-8895-ff39f8febb53-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.921450 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.921425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.922947 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.922902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.923302 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.923285    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/991b8a71-6107-4f4c-8abc-a3136e76f7cb-node-exporter-tls\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.923757 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.923730    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.930366 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.930319    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8929l\" (UniqueName: \"kubernetes.io/projected/991b8a71-6107-4f4c-8abc-a3136e76f7cb-kube-api-access-8929l\") pod \"node-exporter-jlr74\" (UID: \"991b8a71-6107-4f4c-8abc-a3136e76f7cb\") " pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:22.930692 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.930637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd4ld\" (UniqueName: \"kubernetes.io/projected/9d96b046-bc87-4458-8895-ff39f8febb53-kube-api-access-fd4ld\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:22.944893 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:22.944815    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.076127 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.076079    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jlr74"
Jul 28 17:18:23.115927 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.115894    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.231862 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.231828    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.324319 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.324276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b17bc288-c3fc-4fee-8500-f0e61dfd3729-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:23.326957 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.326933    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b17bc288-c3fc-4fee-8500-f0e61dfd3729-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-47z74\" (UID: \"b17bc288-c3fc-4fee-8500-f0e61dfd3729\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:23.425736 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.425703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:23.426234 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:18:23.425866    2583 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 28 17:18:23.426234 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:18:23.425947    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-tls podName:9d96b046-bc87-4458-8895-ff39f8febb53 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:24.425926875 +0000 UTC m=+70.439306850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-fcthp" (UID: "9d96b046-bc87-4458-8895-ff39f8febb53") : secret "kube-state-metrics-tls" not found
Jul 28 17:18:23.442056 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.442005    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.531748 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.531670    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.595951 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.595917    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"
Jul 28 17:18:23.636083 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.636042    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.726113 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.726034    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.730261 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.730231    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:18:23.732787 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.732768    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.735312 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.735261    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 17:18:23.735312 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.735284    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 17:18:23.735312 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.735302    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 17:18:23.735312 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.735266    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bbp8f\""
Jul 28 17:18:23.735587 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.735320    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 17:18:23.735587 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.735300    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 17:18:23.735848 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.735827    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 17:18:23.735848 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.735842    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 17:18:23.736066 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.735827    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 17:18:23.736066 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.735908    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 17:18:23.752449 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.752352    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:18:23.829423 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829356    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77dadaa7-6577-46c0-ac84-002dfb51b086-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829423 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/77dadaa7-6577-46c0-ac84-002dfb51b086-tls-assets\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829423 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-config-volume\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829676 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829478    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49pm8\" (UniqueName: \"kubernetes.io/projected/77dadaa7-6577-46c0-ac84-002dfb51b086-kube-api-access-49pm8\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829676 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-web-config\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829676 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77dadaa7-6577-46c0-ac84-002dfb51b086-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829676 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829676 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829903 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829903 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829903 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829747    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829903 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/77dadaa7-6577-46c0-ac84-002dfb51b086-config-out\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.829903 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.829835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/77dadaa7-6577-46c0-ac84-002dfb51b086-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.848153 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.848124    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.905276 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.905245    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b8ffcbb54-s67n4"
Jul 28 17:18:23.927904 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.927821    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.930923 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.930887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.931078 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.930943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.934119 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.934096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.934453 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.934407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935274 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.930979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935377 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.935304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935377 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.935340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935520 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.935397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/77dadaa7-6577-46c0-ac84-002dfb51b086-config-out\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935520 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.935426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/77dadaa7-6577-46c0-ac84-002dfb51b086-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935520 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.935512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77dadaa7-6577-46c0-ac84-002dfb51b086-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935675 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.935541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/77dadaa7-6577-46c0-ac84-002dfb51b086-tls-assets\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935675 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.935549    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935675 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.935567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-config-volume\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935675 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.935613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49pm8\" (UniqueName: \"kubernetes.io/projected/77dadaa7-6577-46c0-ac84-002dfb51b086-kube-api-access-49pm8\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935675 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.935659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-web-config\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.935945 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.935686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77dadaa7-6577-46c0-ac84-002dfb51b086-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.936687 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.936665    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77dadaa7-6577-46c0-ac84-002dfb51b086-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.937887 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.937474    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77dadaa7-6577-46c0-ac84-002dfb51b086-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.938171 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.938132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/77dadaa7-6577-46c0-ac84-002dfb51b086-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.940366 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.940344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-web-config\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.940894 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.940810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.941302 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.941281    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-config-volume\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.941393 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.941315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/77dadaa7-6577-46c0-ac84-002dfb51b086-config-out\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.941393 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.941326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/77dadaa7-6577-46c0-ac84-002dfb51b086-tls-assets\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.942837 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.942813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77dadaa7-6577-46c0-ac84-002dfb51b086-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:23.945181 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:23.945157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49pm8\" (UniqueName: \"kubernetes.io/projected/77dadaa7-6577-46c0-ac84-002dfb51b086-kube-api-access-49pm8\") pod \"alertmanager-main-0\" (UID: \"77dadaa7-6577-46c0-ac84-002dfb51b086\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:24.029558 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:24.029530    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991b8a71_6107_4f4c_8abc_a3136e76f7cb.slice/crio-b4ef1cb155e9e37b9ac3be408c9a2e4d8a636c4017bc34030e61ade6882ec2f8 WatchSource:0}: Error finding container b4ef1cb155e9e37b9ac3be408c9a2e4d8a636c4017bc34030e61ade6882ec2f8: Status 404 returned error can't find the container with id b4ef1cb155e9e37b9ac3be408c9a2e4d8a636c4017bc34030e61ade6882ec2f8
Jul 28 17:18:24.043583 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.043562    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:24.048430 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.048399    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:24.150703 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.150620    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:24.175045 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.174979    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74"]
Jul 28 17:18:24.179057 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:24.178997    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17bc288_c3fc_4fee_8500_f0e61dfd3729.slice/crio-40e428494d120d4ae4f40000456a06105405c05e73d5522c90625603030f09cc WatchSource:0}: Error finding container 40e428494d120d4ae4f40000456a06105405c05e73d5522c90625603030f09cc: Status 404 returned error can't find the container with id 40e428494d120d4ae4f40000456a06105405c05e73d5522c90625603030f09cc
Jul 28 17:18:24.204494 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.204473    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:18:24.207520 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:24.207490    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77dadaa7_6577_46c0_ac84_002dfb51b086.slice/crio-fbd685a5c4f65a92e233496aa22cceee9cd61124a6438b40ef682aa49ae353d9 WatchSource:0}: Error finding container fbd685a5c4f65a92e233496aa22cceee9cd61124a6438b40ef682aa49ae353d9: Status 404 returned error can't find the container with id fbd685a5c4f65a92e233496aa22cceee9cd61124a6438b40ef682aa49ae353d9
Jul 28 17:18:24.440689 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.440605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:24.443054 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.443011    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d96b046-bc87-4458-8895-ff39f8febb53-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-fcthp\" (UID: \"9d96b046-bc87-4458-8895-ff39f8febb53\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:24.486903 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.486869    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:24.562116 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.562087    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"
Jul 28 17:18:24.582575 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.582547    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:24.680471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.680437    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:24.700610 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.700443    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp"]
Jul 28 17:18:24.703318 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:24.703283    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d96b046_bc87_4458_8895_ff39f8febb53.slice/crio-47d0770e81ed1a10a90491eec46b921297949f2dc855486f51b41cd8f9d90458 WatchSource:0}: Error finding container 47d0770e81ed1a10a90491eec46b921297949f2dc855486f51b41cd8f9d90458: Status 404 returned error can't find the container with id 47d0770e81ed1a10a90491eec46b921297949f2dc855486f51b41cd8f9d90458
Jul 28 17:18:24.998711 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:24.998642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp" event={"ID":"9d96b046-bc87-4458-8895-ff39f8febb53","Type":"ContainerStarted","Data":"47d0770e81ed1a10a90491eec46b921297949f2dc855486f51b41cd8f9d90458"}
Jul 28 17:18:25.000081 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.000054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlr74" event={"ID":"991b8a71-6107-4f4c-8abc-a3136e76f7cb","Type":"ContainerStarted","Data":"b4ef1cb155e9e37b9ac3be408c9a2e4d8a636c4017bc34030e61ade6882ec2f8"}
Jul 28 17:18:25.001845 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.001815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qllgl" event={"ID":"147f1d2b-fe6a-4336-a15c-6362fc89ec8f","Type":"ContainerStarted","Data":"023bb57fe3e21e2063ff71fe1a27d7bd12dc6fa1998155d553cdeae65ab00fc4"}
Jul 28 17:18:25.001987 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.001936    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:18:25.003810 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.003781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-56htm" event={"ID":"b5c24f95-13e0-4739-80ae-b6b128e520b2","Type":"ContainerStarted","Data":"eb00425785c39e42955681532b8bd2fc8d57919289016f9b2913fcf374a26857"}
Jul 28 17:18:25.005080 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.005048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77dadaa7-6577-46c0-ac84-002dfb51b086","Type":"ContainerStarted","Data":"fbd685a5c4f65a92e233496aa22cceee9cd61124a6438b40ef682aa49ae353d9"}
Jul 28 17:18:25.007155 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.007126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74" event={"ID":"b17bc288-c3fc-4fee-8500-f0e61dfd3729","Type":"ContainerStarted","Data":"513e0e5fc33f5635df26a271a1addb4d6baeb6fdd7e8be9e1ffb4847ace7f60d"}
Jul 28 17:18:25.007249 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.007185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74" event={"ID":"b17bc288-c3fc-4fee-8500-f0e61dfd3729","Type":"ContainerStarted","Data":"607bf1c8d280688bb1ee3edc85280aacc82d409f98567e74420af2f09d1f5d33"}
Jul 28 17:18:25.007249 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.007205    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74" event={"ID":"b17bc288-c3fc-4fee-8500-f0e61dfd3729","Type":"ContainerStarted","Data":"40e428494d120d4ae4f40000456a06105405c05e73d5522c90625603030f09cc"}
Jul 28 17:18:25.037145 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.037096    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qllgl" podStartSLOduration=65.542169612 podStartE2EDuration="1m10.037078191s" totalImagesPullingTime="4.494908579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:25.034849758 +0000 UTC m=+71.048229653" watchObservedRunningTime="2026-07-28 17:18:25.037078191 +0000 UTC m=+71.050458085"
Jul 28 17:18:25.069749 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.069700    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-56htm" podStartSLOduration=65.530510154 podStartE2EDuration="1m10.069685123s" totalImagesPullingTime="4.539174969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:25.068079183 +0000 UTC m=+71.081459077" watchObservedRunningTime="2026-07-28 17:18:25.069685123 +0000 UTC m=+71.083065016"
Jul 28 17:18:25.465197 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.465159    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:25.556149 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.556113    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:25.693569 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.692207    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:25.795088 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:25.795035    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:26.014606 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:26.014468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74" event={"ID":"b17bc288-c3fc-4fee-8500-f0e61dfd3729","Type":"ContainerStarted","Data":"c6c6aef3af00e4a875e0c264a12661716931163123555b8f5b4d1e3fba4619ef"}
Jul 28 17:18:26.016566 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:26.016532    2583 generic.go:356] "Generic (PLEG): container finished" podID="991b8a71-6107-4f4c-8abc-a3136e76f7cb" containerID="7de91ca5d82ddf0402c2cac276d3baff64bf682285774d0223f065492c65b76e" exitCode=0
Jul 28 17:18:26.016720 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:26.016589    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlr74" event={"ID":"991b8a71-6107-4f4c-8abc-a3136e76f7cb","Type":"ContainerDied","Data":"7de91ca5d82ddf0402c2cac276d3baff64bf682285774d0223f065492c65b76e"}
Jul 28 17:18:26.042801 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:26.042754    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47z74" podStartSLOduration=3.008382286 podStartE2EDuration="4.042736225s" totalImagesPullingTime="1.034353939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:26.038462247 +0000 UTC m=+72.051842140" watchObservedRunningTime="2026-07-28 17:18:26.042736225 +0000 UTC m=+72.056116112"
Jul 28 17:18:26.246686 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:26.246652    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:26.480392 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:26.480360    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:26.689331 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:26.689247    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:26.754400 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:26.754350    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:27.021425 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.021395    2583 generic.go:356] "Generic (PLEG): container finished" podID="77dadaa7-6577-46c0-ac84-002dfb51b086" containerID="9cf66bc0ffb6328c008e323c1661c0beddd116705717cd9cb3935e44ce7681ed" exitCode=0
Jul 28 17:18:27.021606 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.021480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77dadaa7-6577-46c0-ac84-002dfb51b086","Type":"ContainerDied","Data":"9cf66bc0ffb6328c008e323c1661c0beddd116705717cd9cb3935e44ce7681ed"}
Jul 28 17:18:27.023554 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.023532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp" event={"ID":"9d96b046-bc87-4458-8895-ff39f8febb53","Type":"ContainerStarted","Data":"6e92355dd4fb85ebafb07c3c229c66ac8dc7628cb1a33d2e9cd716159372ac1e"}
Jul 28 17:18:27.023648 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.023591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp" event={"ID":"9d96b046-bc87-4458-8895-ff39f8febb53","Type":"ContainerStarted","Data":"e24ffeb2219fc106ca64420df75a660f429d0d060c09772cb4e217808cab99d6"}
Jul 28 17:18:27.023648 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.023620    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp" event={"ID":"9d96b046-bc87-4458-8895-ff39f8febb53","Type":"ContainerStarted","Data":"85781874f6db4531a90c2b542d580b8f8e62eb6e97f487edefbd45be72d7b17e"}
Jul 28 17:18:27.025703 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.025680    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlr74" event={"ID":"991b8a71-6107-4f4c-8abc-a3136e76f7cb","Type":"ContainerStarted","Data":"5242ab7df6332ba91bf8be7c4db614c314515eef5b16018d5db6c3d64fc67f6d"}
Jul 28 17:18:27.025803 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.025733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlr74" event={"ID":"991b8a71-6107-4f4c-8abc-a3136e76f7cb","Type":"ContainerStarted","Data":"c72398a6b29b7615b63750e1afac95764cb1273ef0f0d76df5c7251652dbe10b"}
Jul 28 17:18:27.078135 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.078091    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-fcthp" podStartSLOduration=3.646427659 podStartE2EDuration="5.07805574s" totalImagesPullingTime="1.431628081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:27.077304915 +0000 UTC m=+73.090684808" watchObservedRunningTime="2026-07-28 17:18:27.07805574 +0000 UTC m=+73.091435632"
Jul 28 17:18:27.097702 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.097644    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jlr74" podStartSLOduration=3.737162526 podStartE2EDuration="5.097626605s" totalImagesPullingTime="1.282398866s" totalInitContainerRuntime="78.065213ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:27.096805445 +0000 UTC m=+73.110185338" watchObservedRunningTime="2026-07-28 17:18:27.097626605 +0000 UTC m=+73.111006499"
Jul 28 17:18:27.417026 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.416930    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:27.486219 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.486187    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:27.611489 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.611449    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:27.881386 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.881350    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"]
Jul 28 17:18:27.884111 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.884088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:27.886762 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.886727    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kqx9s\""
Jul 28 17:18:27.886888 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.886731    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 17:18:27.886888 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.886793    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 17:18:27.886888 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.886875    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 17:18:27.887199 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.887177    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 17:18:27.888329 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.888310    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 17:18:27.891978 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.891957    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 17:18:27.902746 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.902702    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"]
Jul 28 17:18:27.973783 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.973749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:27.973940 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.973799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-federate-client-tls\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:27.973940 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.973829    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j2m9\" (UniqueName: \"kubernetes.io/projected/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-kube-api-access-2j2m9\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:27.973940 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.973856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:27.973940 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.973921    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-serving-certs-ca-bundle\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:27.974149 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.973946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-metrics-client-ca\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:27.974149 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.973970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-secret-telemeter-client\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:27.974149 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:27.973990    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-telemeter-client-tls\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.075145 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.075115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.075303 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.075160    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-federate-client-tls\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.075303 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.075192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j2m9\" (UniqueName: \"kubernetes.io/projected/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-kube-api-access-2j2m9\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.075303 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.075215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.075303 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.075267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-serving-certs-ca-bundle\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.075303 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.075287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-metrics-client-ca\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.075561 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.075312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-secret-telemeter-client\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.075561 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.075336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-telemeter-client-tls\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.076113 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.076084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-serving-certs-ca-bundle\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.076335 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.076308    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-metrics-client-ca\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.076446 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.076393    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.078055 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.078011    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-federate-client-tls\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.078475 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.078452    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.078892 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.078869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-secret-telemeter-client\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.078892 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.078886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-telemeter-client-tls\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.084754 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.084709    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j2m9\" (UniqueName: \"kubernetes.io/projected/d5e4d4de-8e6b-4fc4-95b5-499c557118ae-kube-api-access-2j2m9\") pod \"telemeter-client-7b74b9557b-xkxbv\" (UID: \"d5e4d4de-8e6b-4fc4-95b5-499c557118ae\") " pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.196604 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.196527    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"
Jul 28 17:18:28.480748 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.480710    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:28.536472 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:18:28.536442    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e4d4de_8e6b_4fc4_95b5_499c557118ae.slice/crio-609a8daec22d59274f9312ab8e4a0cd8db85606c98c4c23375f31b8027b79d2f WatchSource:0}: Error finding container 609a8daec22d59274f9312ab8e4a0cd8db85606c98c4c23375f31b8027b79d2f: Status 404 returned error can't find the container with id 609a8daec22d59274f9312ab8e4a0cd8db85606c98c4c23375f31b8027b79d2f
Jul 28 17:18:28.539162 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.539143    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b74b9557b-xkxbv"]
Jul 28 17:18:28.584364 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.584333    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:28.872065 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:28.872002    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:29.038182 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:29.038144    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77dadaa7-6577-46c0-ac84-002dfb51b086","Type":"ContainerStarted","Data":"1dee6c50c0f1bf2afcc41bb75d3d99de7ac48fcb57374b201864eeb93427d5e2"}
Jul 28 17:18:29.038327 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:29.038230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77dadaa7-6577-46c0-ac84-002dfb51b086","Type":"ContainerStarted","Data":"e24dda6d05f2f83a36b31ad5a192a6b181275db39fb760bf929497e255dd0db7"}
Jul 28 17:18:29.038327 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:29.038276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77dadaa7-6577-46c0-ac84-002dfb51b086","Type":"ContainerStarted","Data":"7f29f91d3fb564045e8b105d913de03181372ba0693bafdf511ec5a9a44e36d4"}
Jul 28 17:18:29.038327 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:29.038315    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77dadaa7-6577-46c0-ac84-002dfb51b086","Type":"ContainerStarted","Data":"c7b886514e58c7fbd64afe5df600f04b39e7e9868cd86402d7d6d742bc87c42c"}
Jul 28 17:18:29.038508 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:29.038352    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77dadaa7-6577-46c0-ac84-002dfb51b086","Type":"ContainerStarted","Data":"d6999663b255354df5fa22f2c57923eec14dc2a6d0d63dda81a9d8ef87c6ac1a"}
Jul 28 17:18:29.039320 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:29.039298    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv" event={"ID":"d5e4d4de-8e6b-4fc4-95b5-499c557118ae","Type":"ContainerStarted","Data":"609a8daec22d59274f9312ab8e4a0cd8db85606c98c4c23375f31b8027b79d2f"}
Jul 28 17:18:29.108014 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:29.107856    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:29.199492 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:29.199454    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:30.048205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:30.048116    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77dadaa7-6577-46c0-ac84-002dfb51b086","Type":"ContainerStarted","Data":"668b38d88f2aa68185e8e230289512610ab55ec80c00efc6a71f2fa420b459ca"}
Jul 28 17:18:30.077596 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:30.077528    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.347763024 podStartE2EDuration="7.07750567s" totalImagesPullingTime="3.706531317s" totalInitContainerRuntime="23.211329ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:30.075450104 +0000 UTC m=+76.088829996" watchObservedRunningTime="2026-07-28 17:18:30.07750567 +0000 UTC m=+76.090885564"
Jul 28 17:18:31.053585 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:31.053545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv" event={"ID":"d5e4d4de-8e6b-4fc4-95b5-499c557118ae","Type":"ContainerStarted","Data":"cc8267eb2622aa66926b496bde27924dc1420bf7eb0ceb40de7fb4292ff506c2"}
Jul 28 17:18:31.054057 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:31.053614    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv" event={"ID":"d5e4d4de-8e6b-4fc4-95b5-499c557118ae","Type":"ContainerStarted","Data":"3fb97dac9bc818d0913bb57d2c1b5f41d6a841079f3ddcfbf01c038aadbc67d3"}
Jul 28 17:18:31.054057 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:31.053645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv" event={"ID":"d5e4d4de-8e6b-4fc4-95b5-499c557118ae","Type":"ContainerStarted","Data":"32a8295ed5a093ad8f94d2b502df2f8fe09aabdda57cabdfc9d4a8db9c5bb0bf"}
Jul 28 17:18:31.080076 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:31.080008    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7b74b9557b-xkxbv" podStartSLOduration=2.197414467 podStartE2EDuration="4.079991488s" totalImagesPullingTime="1.882577021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:31.078116251 +0000 UTC m=+77.091496145" watchObservedRunningTime="2026-07-28 17:18:31.079991488 +0000 UTC m=+77.093371380"
Jul 28 17:18:32.295609 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:32.295576    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:32.425992 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:32.425945    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:34.583687 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:34.583649    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:34.672098 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:34.672063    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:38.093295 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:38.093173    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:38.194647 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:38.194613    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:42.808868 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:42.808818    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:42.917369 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:42.917338    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:43.709405 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:43.709369    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:43.772260 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:43.772226    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:50.384453 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:50.384414    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:56.019343 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:18:56.019315    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qllgl"
Jul 28 17:19:00.754573 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:00.754536    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:00.816112 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:00.816068    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:38.896460 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:38.896420    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:38.995294 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:38.995255    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:45.463690 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:45.463658    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:45.535846 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:45.535816    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:50.792453 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:50.792411    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:50.907198 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:50.907159    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:56.122449 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:56.122413    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:56.228303 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:56.228261    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:56.289031 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:56.288998    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:58.720184 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:58.720144    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:58.821126 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:19:58.821081    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:30.612388 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:30.612345    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:34.593626 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:34.593591    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:34.653201 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:34.653150    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:55.610148 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:55.610105    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:55.708187 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:55.708146    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:56.009701 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:56.009655    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:56.109993 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:56.109959    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:56.248281 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:56.248247    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:58.418293 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.418257    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"]
Jul 28 17:20:58.420698 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.420676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:20:58.423402 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.423383    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p2nq6\""
Jul 28 17:20:58.423565 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.423550    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 17:20:58.423644 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.423553    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 17:20:58.431171 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.431152    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"]
Jul 28 17:20:58.556372 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.556337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:20:58.556372 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.556375    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:20:58.556613 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.556399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zc22\" (UniqueName: \"kubernetes.io/projected/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-kube-api-access-7zc22\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:20:58.657893 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.657857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:20:58.657893 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.657898    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zc22\" (UniqueName: \"kubernetes.io/projected/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-kube-api-access-7zc22\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:20:58.658139 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.657985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:20:58.658329 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.658311    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:20:58.658367 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.658342    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:20:58.666533 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.666508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zc22\" (UniqueName: \"kubernetes.io/projected/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-kube-api-access-7zc22\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:20:58.730141 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.730111    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:20:58.856455 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:58.856423    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"]
Jul 28 17:20:58.859342 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:20:58.859314    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc92b1a2_9077_47b3_9b4b_906eda3ea94c.slice/crio-6d0a2087d7d10f6a213d1effe4cd63f258ddc32461d4b7d6a6b894529af679ef WatchSource:0}: Error finding container 6d0a2087d7d10f6a213d1effe4cd63f258ddc32461d4b7d6a6b894529af679ef: Status 404 returned error can't find the container with id 6d0a2087d7d10f6a213d1effe4cd63f258ddc32461d4b7d6a6b894529af679ef
Jul 28 17:20:59.579881 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:20:59.579844    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7" event={"ID":"cc92b1a2-9077-47b3-9b4b-906eda3ea94c","Type":"ContainerStarted","Data":"6d0a2087d7d10f6a213d1effe4cd63f258ddc32461d4b7d6a6b894529af679ef"}
Jul 28 17:21:05.603422 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:05.603382    2583 generic.go:356] "Generic (PLEG): container finished" podID="cc92b1a2-9077-47b3-9b4b-906eda3ea94c" containerID="7b963767100aa815b36b101b41921363a49a7d225d807d901cde0a7837ffb96b" exitCode=0
Jul 28 17:21:05.603904 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:05.603429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7" event={"ID":"cc92b1a2-9077-47b3-9b4b-906eda3ea94c","Type":"ContainerDied","Data":"7b963767100aa815b36b101b41921363a49a7d225d807d901cde0a7837ffb96b"}
Jul 28 17:21:08.615754 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:08.615716    2583 generic.go:356] "Generic (PLEG): container finished" podID="cc92b1a2-9077-47b3-9b4b-906eda3ea94c" containerID="86f828147119004940b363a643b17e81ce6e1d546bf496aee5235690c7603abd" exitCode=0
Jul 28 17:21:08.616138 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:08.615781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7" event={"ID":"cc92b1a2-9077-47b3-9b4b-906eda3ea94c","Type":"ContainerDied","Data":"86f828147119004940b363a643b17e81ce6e1d546bf496aee5235690c7603abd"}
Jul 28 17:21:16.650388 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:16.650358    2583 generic.go:356] "Generic (PLEG): container finished" podID="cc92b1a2-9077-47b3-9b4b-906eda3ea94c" containerID="96aa800bfeb4fbc0999f41bccb2529ebb67c220111439c71d79eda0c725bed3d" exitCode=0
Jul 28 17:21:16.650731 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:16.650441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7" event={"ID":"cc92b1a2-9077-47b3-9b4b-906eda3ea94c","Type":"ContainerDied","Data":"96aa800bfeb4fbc0999f41bccb2529ebb67c220111439c71d79eda0c725bed3d"}
Jul 28 17:21:17.779618 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:17.779592    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:21:17.812670 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:17.812648    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-util\") pod \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") "
Jul 28 17:21:17.812803 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:17.812690    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-bundle\") pod \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") "
Jul 28 17:21:17.812803 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:17.812724    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zc22\" (UniqueName: \"kubernetes.io/projected/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-kube-api-access-7zc22\") pod \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\" (UID: \"cc92b1a2-9077-47b3-9b4b-906eda3ea94c\") "
Jul 28 17:21:17.813467 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:17.813436    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-bundle" (OuterVolumeSpecName: "bundle") pod "cc92b1a2-9077-47b3-9b4b-906eda3ea94c" (UID: "cc92b1a2-9077-47b3-9b4b-906eda3ea94c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:21:17.814942 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:17.814920    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-kube-api-access-7zc22" (OuterVolumeSpecName: "kube-api-access-7zc22") pod "cc92b1a2-9077-47b3-9b4b-906eda3ea94c" (UID: "cc92b1a2-9077-47b3-9b4b-906eda3ea94c"). InnerVolumeSpecName "kube-api-access-7zc22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:21:17.817477 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:17.817457    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-util" (OuterVolumeSpecName: "util") pod "cc92b1a2-9077-47b3-9b4b-906eda3ea94c" (UID: "cc92b1a2-9077-47b3-9b4b-906eda3ea94c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:21:17.914208 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:17.914132    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-util\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:21:17.914208 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:17.914158    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-bundle\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:21:17.914208 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:17.914167    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zc22\" (UniqueName: \"kubernetes.io/projected/cc92b1a2-9077-47b3-9b4b-906eda3ea94c-kube-api-access-7zc22\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:21:18.659660 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:18.659629    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7"
Jul 28 17:21:18.659660 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:18.659644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kf7c7" event={"ID":"cc92b1a2-9077-47b3-9b4b-906eda3ea94c","Type":"ContainerDied","Data":"6d0a2087d7d10f6a213d1effe4cd63f258ddc32461d4b7d6a6b894529af679ef"}
Jul 28 17:21:18.659851 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:18.659696    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d0a2087d7d10f6a213d1effe4cd63f258ddc32461d4b7d6a6b894529af679ef"
Jul 28 17:21:20.625303 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:20.625264    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:20.719681 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:20.719651    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:22.058944 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.058911    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"]
Jul 28 17:21:22.059344 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.059190    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc92b1a2-9077-47b3-9b4b-906eda3ea94c" containerName="pull"
Jul 28 17:21:22.059344 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.059200    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc92b1a2-9077-47b3-9b4b-906eda3ea94c" containerName="pull"
Jul 28 17:21:22.059344 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.059216    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc92b1a2-9077-47b3-9b4b-906eda3ea94c" containerName="util"
Jul 28 17:21:22.059344 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.059221    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc92b1a2-9077-47b3-9b4b-906eda3ea94c" containerName="util"
Jul 28 17:21:22.059344 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.059236    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc92b1a2-9077-47b3-9b4b-906eda3ea94c" containerName="extract"
Jul 28 17:21:22.059344 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.059241    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc92b1a2-9077-47b3-9b4b-906eda3ea94c" containerName="extract"
Jul 28 17:21:22.059344 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.059282    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc92b1a2-9077-47b3-9b4b-906eda3ea94c" containerName="extract"
Jul 28 17:21:22.062495 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.062478    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"
Jul 28 17:21:22.065068 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.065039    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-4vj9s\""
Jul 28 17:21:22.065197 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.065075    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 17:21:22.065197 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.065126    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 17:21:22.065197 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.065141    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 17:21:22.075257 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.075234    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"]
Jul 28 17:21:22.146942 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.146907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5258794a-2d42-41b0-9616-112c347840ac-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp\" (UID: \"5258794a-2d42-41b0-9616-112c347840ac\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"
Jul 28 17:21:22.147136 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.146952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47gw6\" (UniqueName: \"kubernetes.io/projected/5258794a-2d42-41b0-9616-112c347840ac-kube-api-access-47gw6\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp\" (UID: \"5258794a-2d42-41b0-9616-112c347840ac\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"
Jul 28 17:21:22.248297 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.248262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5258794a-2d42-41b0-9616-112c347840ac-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp\" (UID: \"5258794a-2d42-41b0-9616-112c347840ac\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"
Jul 28 17:21:22.248504 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.248307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47gw6\" (UniqueName: \"kubernetes.io/projected/5258794a-2d42-41b0-9616-112c347840ac-kube-api-access-47gw6\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp\" (UID: \"5258794a-2d42-41b0-9616-112c347840ac\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"
Jul 28 17:21:22.250533 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.250512    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5258794a-2d42-41b0-9616-112c347840ac-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp\" (UID: \"5258794a-2d42-41b0-9616-112c347840ac\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"
Jul 28 17:21:22.257922 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.257898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47gw6\" (UniqueName: \"kubernetes.io/projected/5258794a-2d42-41b0-9616-112c347840ac-kube-api-access-47gw6\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp\" (UID: \"5258794a-2d42-41b0-9616-112c347840ac\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"
Jul 28 17:21:22.371934 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.371851    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"
Jul 28 17:21:22.503996 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.503927    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"]
Jul 28 17:21:22.506417 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:21:22.506376    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5258794a_2d42_41b0_9616_112c347840ac.slice/crio-c698938736eb9b9aff73081fb82bd5307135477137693bf5ca7a9364dffab698 WatchSource:0}: Error finding container c698938736eb9b9aff73081fb82bd5307135477137693bf5ca7a9364dffab698: Status 404 returned error can't find the container with id c698938736eb9b9aff73081fb82bd5307135477137693bf5ca7a9364dffab698
Jul 28 17:21:22.676217 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:22.676135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp" event={"ID":"5258794a-2d42-41b0-9616-112c347840ac","Type":"ContainerStarted","Data":"c698938736eb9b9aff73081fb82bd5307135477137693bf5ca7a9364dffab698"}
Jul 28 17:21:23.669955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:23.669918    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:23.767215 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:23.767177    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:25.691215 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:25.691182    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp" event={"ID":"5258794a-2d42-41b0-9616-112c347840ac","Type":"ContainerStarted","Data":"0066fc5365389d2407bf3088348731e954271fdfcdbd8670d5b8165ee63ffc48"}
Jul 28 17:21:25.691591 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:25.691326    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"
Jul 28 17:21:25.713798 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:25.713748    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp" podStartSLOduration=0.746523826 podStartE2EDuration="3.713735671s" totalImagesPullingTime="2.967211845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:25.712582435 +0000 UTC m=+251.725962338" watchObservedRunningTime="2026-07-28 17:21:25.713735671 +0000 UTC m=+251.727115564"
Jul 28 17:21:25.995110 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:25.995071    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:26.251918 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.251834    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:26.269133 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.269103    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-9hrmw"]
Jul 28 17:21:26.272131 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.272113    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:26.274512 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.274492    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 17:21:26.274625 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.274585    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 17:21:26.274741 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.274725    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hgqhb\""
Jul 28 17:21:26.283326 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.283302    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-9hrmw"]
Jul 28 17:21:26.382343 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.382312    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8xlg\" (UniqueName: \"kubernetes.io/projected/56bf1dbe-3afe-42c7-8fec-fba48f025bba-kube-api-access-h8xlg\") pod \"keda-operator-77f648d694-9hrmw\" (UID: \"56bf1dbe-3afe-42c7-8fec-fba48f025bba\") " pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:26.382502 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.382358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/56bf1dbe-3afe-42c7-8fec-fba48f025bba-certificates\") pod \"keda-operator-77f648d694-9hrmw\" (UID: \"56bf1dbe-3afe-42c7-8fec-fba48f025bba\") " pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:26.382502 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.382410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/56bf1dbe-3afe-42c7-8fec-fba48f025bba-cabundle0\") pod \"keda-operator-77f648d694-9hrmw\" (UID: \"56bf1dbe-3afe-42c7-8fec-fba48f025bba\") " pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:26.483147 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.483118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/56bf1dbe-3afe-42c7-8fec-fba48f025bba-certificates\") pod \"keda-operator-77f648d694-9hrmw\" (UID: \"56bf1dbe-3afe-42c7-8fec-fba48f025bba\") " pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:26.483330 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.483159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/56bf1dbe-3afe-42c7-8fec-fba48f025bba-cabundle0\") pod \"keda-operator-77f648d694-9hrmw\" (UID: \"56bf1dbe-3afe-42c7-8fec-fba48f025bba\") " pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:26.483330 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.483202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8xlg\" (UniqueName: \"kubernetes.io/projected/56bf1dbe-3afe-42c7-8fec-fba48f025bba-kube-api-access-h8xlg\") pod \"keda-operator-77f648d694-9hrmw\" (UID: \"56bf1dbe-3afe-42c7-8fec-fba48f025bba\") " pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:26.483330 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:21:26.483302    2583 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 17:21:26.483330 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:21:26.483326    2583 secret.go:281] references non-existent secret key: ca.crt
Jul 28 17:21:26.483541 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:21:26.483337    2583 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 17:21:26.483541 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:21:26.483353    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-9hrmw: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 17:21:26.483541 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:21:26.483432    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56bf1dbe-3afe-42c7-8fec-fba48f025bba-certificates podName:56bf1dbe-3afe-42c7-8fec-fba48f025bba nodeName:}" failed. No retries permitted until 2026-07-28 17:21:26.983410783 +0000 UTC m=+252.996790672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/56bf1dbe-3afe-42c7-8fec-fba48f025bba-certificates") pod "keda-operator-77f648d694-9hrmw" (UID: "56bf1dbe-3afe-42c7-8fec-fba48f025bba") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 17:21:26.483801 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.483784    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/56bf1dbe-3afe-42c7-8fec-fba48f025bba-cabundle0\") pod \"keda-operator-77f648d694-9hrmw\" (UID: \"56bf1dbe-3afe-42c7-8fec-fba48f025bba\") " pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:26.492451 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.492428    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8xlg\" (UniqueName: \"kubernetes.io/projected/56bf1dbe-3afe-42c7-8fec-fba48f025bba-kube-api-access-h8xlg\") pod \"keda-operator-77f648d694-9hrmw\" (UID: \"56bf1dbe-3afe-42c7-8fec-fba48f025bba\") " pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:26.497625 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.497596    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:26.596614 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.596530    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:26.698884 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.698852    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"]
Jul 28 17:21:26.701927 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.701911    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:26.704382 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.704361    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 17:21:26.712642 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.712612    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"]
Jul 28 17:21:26.786657 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.786621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-certificates\") pod \"keda-metrics-apiserver-55557df8c-n2n5d\" (UID: \"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:26.786839 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.786687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwt6\" (UniqueName: \"kubernetes.io/projected/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-kube-api-access-9lwt6\") pod \"keda-metrics-apiserver-55557df8c-n2n5d\" (UID: \"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:26.786839 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.786795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-n2n5d\" (UID: \"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:26.888447 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.888352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-certificates\") pod \"keda-metrics-apiserver-55557df8c-n2n5d\" (UID: \"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:26.888447 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.888439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwt6\" (UniqueName: \"kubernetes.io/projected/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-kube-api-access-9lwt6\") pod \"keda-metrics-apiserver-55557df8c-n2n5d\" (UID: \"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:26.888647 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.888500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-n2n5d\" (UID: \"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:26.888647 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:21:26.888519    2583 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 28 17:21:26.888647 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:21:26.888548    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d: secret "keda-metrics-apiserver-certs" not found
Jul 28 17:21:26.888647 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:21:26.888606    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-certificates podName:6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8 nodeName:}" failed. No retries permitted until 2026-07-28 17:21:27.388590998 +0000 UTC m=+253.401970868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-certificates") pod "keda-metrics-apiserver-55557df8c-n2n5d" (UID: "6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8") : secret "keda-metrics-apiserver-certs" not found
Jul 28 17:21:26.888891 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.888873    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-n2n5d\" (UID: \"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:26.899778 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.899746    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:26.902208 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.902181    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwt6\" (UniqueName: \"kubernetes.io/projected/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-kube-api-access-9lwt6\") pod \"keda-metrics-apiserver-55557df8c-n2n5d\" (UID: \"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:26.907450 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.907429    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-t4njz"]
Jul 28 17:21:26.911976 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.911960    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-t4njz"
Jul 28 17:21:26.914371 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.914352    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 17:21:26.919862 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.919838    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-t4njz"]
Jul 28 17:21:26.989450 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.989416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/56bf1dbe-3afe-42c7-8fec-fba48f025bba-certificates\") pod \"keda-operator-77f648d694-9hrmw\" (UID: \"56bf1dbe-3afe-42c7-8fec-fba48f025bba\") " pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:26.989623 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.989482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/04324c5c-143b-4bd6-906e-c3b2a5432b76-certificates\") pod \"keda-admission-56f9798bb4-t4njz\" (UID: \"04324c5c-143b-4bd6-906e-c3b2a5432b76\") " pod="openshift-keda/keda-admission-56f9798bb4-t4njz"
Jul 28 17:21:26.989623 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.989513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dljk\" (UniqueName: \"kubernetes.io/projected/04324c5c-143b-4bd6-906e-c3b2a5432b76-kube-api-access-2dljk\") pod \"keda-admission-56f9798bb4-t4njz\" (UID: \"04324c5c-143b-4bd6-906e-c3b2a5432b76\") " pod="openshift-keda/keda-admission-56f9798bb4-t4njz"
Jul 28 17:21:26.991838 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.991817    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/56bf1dbe-3afe-42c7-8fec-fba48f025bba-certificates\") pod \"keda-operator-77f648d694-9hrmw\" (UID: \"56bf1dbe-3afe-42c7-8fec-fba48f025bba\") " pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:26.994498 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:26.994464    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:27.090581 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.090539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/04324c5c-143b-4bd6-906e-c3b2a5432b76-certificates\") pod \"keda-admission-56f9798bb4-t4njz\" (UID: \"04324c5c-143b-4bd6-906e-c3b2a5432b76\") " pod="openshift-keda/keda-admission-56f9798bb4-t4njz"
Jul 28 17:21:27.090581 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.090585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dljk\" (UniqueName: \"kubernetes.io/projected/04324c5c-143b-4bd6-906e-c3b2a5432b76-kube-api-access-2dljk\") pod \"keda-admission-56f9798bb4-t4njz\" (UID: \"04324c5c-143b-4bd6-906e-c3b2a5432b76\") " pod="openshift-keda/keda-admission-56f9798bb4-t4njz"
Jul 28 17:21:27.090816 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:21:27.090689    2583 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 28 17:21:27.090816 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:21:27.090713    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-t4njz: secret "keda-admission-webhooks-certs" not found
Jul 28 17:21:27.090816 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:21:27.090768    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04324c5c-143b-4bd6-906e-c3b2a5432b76-certificates podName:04324c5c-143b-4bd6-906e-c3b2a5432b76 nodeName:}" failed. No retries permitted until 2026-07-28 17:21:27.590751916 +0000 UTC m=+253.604131788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/04324c5c-143b-4bd6-906e-c3b2a5432b76-certificates") pod "keda-admission-56f9798bb4-t4njz" (UID: "04324c5c-143b-4bd6-906e-c3b2a5432b76") : secret "keda-admission-webhooks-certs" not found
Jul 28 17:21:27.099903 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.099878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dljk\" (UniqueName: \"kubernetes.io/projected/04324c5c-143b-4bd6-906e-c3b2a5432b76-kube-api-access-2dljk\") pod \"keda-admission-56f9798bb4-t4njz\" (UID: \"04324c5c-143b-4bd6-906e-c3b2a5432b76\") " pod="openshift-keda/keda-admission-56f9798bb4-t4njz"
Jul 28 17:21:27.115389 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.115360    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:27.182267 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.182185    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:27.218793 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.218751    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:27.317677 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.317631    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-9hrmw"]
Jul 28 17:21:27.320175 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:21:27.320149    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bf1dbe_3afe_42c7_8fec_fba48f025bba.slice/crio-246fc8516bae860017d16b1cc4d217ac3f920bf1802e982950b66f1af97de818 WatchSource:0}: Error finding container 246fc8516bae860017d16b1cc4d217ac3f920bf1802e982950b66f1af97de818: Status 404 returned error can't find the container with id 246fc8516bae860017d16b1cc4d217ac3f920bf1802e982950b66f1af97de818
Jul 28 17:21:27.328440 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.328413    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:27.393254 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.393219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-certificates\") pod \"keda-metrics-apiserver-55557df8c-n2n5d\" (UID: \"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:27.395620 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.395591    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8-certificates\") pod \"keda-metrics-apiserver-55557df8c-n2n5d\" (UID: \"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:27.595344 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.595310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/04324c5c-143b-4bd6-906e-c3b2a5432b76-certificates\") pod \"keda-admission-56f9798bb4-t4njz\" (UID: \"04324c5c-143b-4bd6-906e-c3b2a5432b76\") " pod="openshift-keda/keda-admission-56f9798bb4-t4njz"
Jul 28 17:21:27.597960 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.597925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/04324c5c-143b-4bd6-906e-c3b2a5432b76-certificates\") pod \"keda-admission-56f9798bb4-t4njz\" (UID: \"04324c5c-143b-4bd6-906e-c3b2a5432b76\") " pod="openshift-keda/keda-admission-56f9798bb4-t4njz"
Jul 28 17:21:27.612678 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.612652    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:27.617234 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.617205    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:27.701132 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.701098    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-9hrmw" event={"ID":"56bf1dbe-3afe-42c7-8fec-fba48f025bba","Type":"ContainerStarted","Data":"246fc8516bae860017d16b1cc4d217ac3f920bf1802e982950b66f1af97de818"}
Jul 28 17:21:27.718729 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.718687    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:27.741723 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.741689    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"]
Jul 28 17:21:27.744742 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:21:27.744714    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9adcfb_1ea5_46f7_9cb7_f1c5e05403b8.slice/crio-4ec001c759d7c62701f0d5976f035ae3ee1e431bf8b97a296791f822ac017645 WatchSource:0}: Error finding container 4ec001c759d7c62701f0d5976f035ae3ee1e431bf8b97a296791f822ac017645: Status 404 returned error can't find the container with id 4ec001c759d7c62701f0d5976f035ae3ee1e431bf8b97a296791f822ac017645
Jul 28 17:21:27.824180 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.824149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-t4njz"
Jul 28 17:21:27.953645 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:27.953614    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-t4njz"]
Jul 28 17:21:27.956673 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:21:27.956644    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04324c5c_143b_4bd6_906e_c3b2a5432b76.slice/crio-eedd0b0fa180aa28891908bfddc5324e921bda6e934b1114468b21a60b85ffc1 WatchSource:0}: Error finding container eedd0b0fa180aa28891908bfddc5324e921bda6e934b1114468b21a60b85ffc1: Status 404 returned error can't find the container with id eedd0b0fa180aa28891908bfddc5324e921bda6e934b1114468b21a60b85ffc1
Jul 28 17:21:28.707493 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:28.707455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-t4njz" event={"ID":"04324c5c-143b-4bd6-906e-c3b2a5432b76","Type":"ContainerStarted","Data":"eedd0b0fa180aa28891908bfddc5324e921bda6e934b1114468b21a60b85ffc1"}
Jul 28 17:21:28.708745 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:28.708715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d" event={"ID":"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8","Type":"ContainerStarted","Data":"4ec001c759d7c62701f0d5976f035ae3ee1e431bf8b97a296791f822ac017645"}
Jul 28 17:21:32.730727 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:32.730685    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-t4njz" event={"ID":"04324c5c-143b-4bd6-906e-c3b2a5432b76","Type":"ContainerStarted","Data":"a516fca012a1f16e605debcbcdf7ccb7251b9ffaab410927b54d0807f80da283"}
Jul 28 17:21:32.731225 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:32.730846    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-t4njz"
Jul 28 17:21:32.732277 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:32.732242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-9hrmw" event={"ID":"56bf1dbe-3afe-42c7-8fec-fba48f025bba","Type":"ContainerStarted","Data":"83bd4a4eb16368a478bc7f8a1a8f77136b811e90a5c464729e92cc6266b38674"}
Jul 28 17:21:32.732413 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:32.732373    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:32.733597 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:32.733576    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d" event={"ID":"6a9adcfb-1ea5-46f7-9cb7-f1c5e05403b8","Type":"ContainerStarted","Data":"5aaa7027a2f1cc52caf74b199bed84049d9988ab08a72079a60898efe8d6f718"}
Jul 28 17:21:32.733721 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:32.733706    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:32.750337 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:32.750292    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-t4njz" podStartSLOduration=2.706464613 podStartE2EDuration="6.750278722s" totalImagesPullingTime="4.043814109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:32.749282071 +0000 UTC m=+258.762661966" watchObservedRunningTime="2026-07-28 17:21:32.750278722 +0000 UTC m=+258.763658615"
Jul 28 17:21:32.766914 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:32.766864    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d" podStartSLOduration=2.504152846 podStartE2EDuration="6.766849871s" totalImagesPullingTime="4.262697025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:32.765666341 +0000 UTC m=+258.779046260" watchObservedRunningTime="2026-07-28 17:21:32.766849871 +0000 UTC m=+258.780229764"
Jul 28 17:21:32.783195 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:32.783147    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-9hrmw" podStartSLOduration=2.095780644 podStartE2EDuration="6.783131368s" totalImagesPullingTime="4.687350724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:32.781790649 +0000 UTC m=+258.795170544" watchObservedRunningTime="2026-07-28 17:21:32.783131368 +0000 UTC m=+258.796511260"
Jul 28 17:21:36.073383 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:36.073278    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:36.131330 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:36.131293    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:43.742195 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:43.742162    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-n2n5d"
Jul 28 17:21:46.697606 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:46.697578    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-7n6fp"
Jul 28 17:21:53.740907 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:53.740868    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-9hrmw"
Jul 28 17:21:53.741391 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:21:53.740951    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-t4njz"
Jul 28 17:22:14.363962 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:14.363930    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:22:14.373153 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:14.373125    2583 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:22:14.374953 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:14.374931    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:22:38.782385 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:38.782348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:40.203484 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:40.203445    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:43.718130 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.718095    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:43.810234 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.810164    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:43.814495 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.814459    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"]
Jul 28 17:22:43.818178 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.818156    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:22:43.820622 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.820597    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 17:22:43.821537 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.821524    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-8f4tj\""
Jul 28 17:22:43.821996 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.821983    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:22:43.822712 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.822695    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:22:43.828211 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.828192    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"]
Jul 28 17:22:43.832101 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.832081    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-gw7kp"]
Jul 28 17:22:43.834197 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.834176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp"
Jul 28 17:22:43.837237 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.837215    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 17:22:43.837442 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.837428    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-54hpn\""
Jul 28 17:22:43.847909 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.847886    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-gw7kp"]
Jul 28 17:22:43.881628 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.881592    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:43.931317 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.931288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmqx\" (UniqueName: \"kubernetes.io/projected/994c5f04-9e59-482e-8094-9e47898d2ca1-kube-api-access-wlmqx\") pod \"kserve-controller-manager-85d8b8ddfc-4rjpq\" (UID: \"994c5f04-9e59-482e-8094-9e47898d2ca1\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:22:43.931499 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.931355    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6skc\" (UniqueName: \"kubernetes.io/projected/9791465e-d449-49bf-8f13-7fb2253d53bf-kube-api-access-q6skc\") pod \"llmisvc-controller-manager-84fc959947-gw7kp\" (UID: \"9791465e-d449-49bf-8f13-7fb2253d53bf\") " pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp"
Jul 28 17:22:43.931499 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.931392    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/994c5f04-9e59-482e-8094-9e47898d2ca1-cert\") pod \"kserve-controller-manager-85d8b8ddfc-4rjpq\" (UID: \"994c5f04-9e59-482e-8094-9e47898d2ca1\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:22:43.931499 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.931432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9791465e-d449-49bf-8f13-7fb2253d53bf-cert\") pod \"llmisvc-controller-manager-84fc959947-gw7kp\" (UID: \"9791465e-d449-49bf-8f13-7fb2253d53bf\") " pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp"
Jul 28 17:22:43.985929 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:43.985892    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:44.032443 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.032414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6skc\" (UniqueName: \"kubernetes.io/projected/9791465e-d449-49bf-8f13-7fb2253d53bf-kube-api-access-q6skc\") pod \"llmisvc-controller-manager-84fc959947-gw7kp\" (UID: \"9791465e-d449-49bf-8f13-7fb2253d53bf\") " pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp"
Jul 28 17:22:44.032600 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.032456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/994c5f04-9e59-482e-8094-9e47898d2ca1-cert\") pod \"kserve-controller-manager-85d8b8ddfc-4rjpq\" (UID: \"994c5f04-9e59-482e-8094-9e47898d2ca1\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:22:44.032600 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.032503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9791465e-d449-49bf-8f13-7fb2253d53bf-cert\") pod \"llmisvc-controller-manager-84fc959947-gw7kp\" (UID: \"9791465e-d449-49bf-8f13-7fb2253d53bf\") " pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp"
Jul 28 17:22:44.032600 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.032541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmqx\" (UniqueName: \"kubernetes.io/projected/994c5f04-9e59-482e-8094-9e47898d2ca1-kube-api-access-wlmqx\") pod \"kserve-controller-manager-85d8b8ddfc-4rjpq\" (UID: \"994c5f04-9e59-482e-8094-9e47898d2ca1\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:22:44.032757 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:22:44.032639    2583 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 28 17:22:44.032757 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:22:44.032698    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9791465e-d449-49bf-8f13-7fb2253d53bf-cert podName:9791465e-d449-49bf-8f13-7fb2253d53bf nodeName:}" failed. No retries permitted until 2026-07-28 17:22:44.532682539 +0000 UTC m=+330.546062414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9791465e-d449-49bf-8f13-7fb2253d53bf-cert") pod "llmisvc-controller-manager-84fc959947-gw7kp" (UID: "9791465e-d449-49bf-8f13-7fb2253d53bf") : secret "llmisvc-webhook-server-cert" not found
Jul 28 17:22:44.034930 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.034908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/994c5f04-9e59-482e-8094-9e47898d2ca1-cert\") pod \"kserve-controller-manager-85d8b8ddfc-4rjpq\" (UID: \"994c5f04-9e59-482e-8094-9e47898d2ca1\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:22:44.052122 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.052093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmqx\" (UniqueName: \"kubernetes.io/projected/994c5f04-9e59-482e-8094-9e47898d2ca1-kube-api-access-wlmqx\") pod \"kserve-controller-manager-85d8b8ddfc-4rjpq\" (UID: \"994c5f04-9e59-482e-8094-9e47898d2ca1\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:22:44.053843 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.053826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6skc\" (UniqueName: \"kubernetes.io/projected/9791465e-d449-49bf-8f13-7fb2253d53bf-kube-api-access-q6skc\") pod \"llmisvc-controller-manager-84fc959947-gw7kp\" (UID: \"9791465e-d449-49bf-8f13-7fb2253d53bf\") " pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp"
Jul 28 17:22:44.106182 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.106148    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:44.129062 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.129008    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:22:44.205039 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.204990    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:44.257303 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.257226    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"]
Jul 28 17:22:44.259865 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:22:44.259844    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994c5f04_9e59_482e_8094_9e47898d2ca1.slice/crio-b2233ffecc35e570d349816c6c6cce574de7b43ccd636aaa4959b68ffde2cc5e WatchSource:0}: Error finding container b2233ffecc35e570d349816c6c6cce574de7b43ccd636aaa4959b68ffde2cc5e: Status 404 returned error can't find the container with id b2233ffecc35e570d349816c6c6cce574de7b43ccd636aaa4959b68ffde2cc5e
Jul 28 17:22:44.261205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.261187    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:22:44.537006 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.536915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9791465e-d449-49bf-8f13-7fb2253d53bf-cert\") pod \"llmisvc-controller-manager-84fc959947-gw7kp\" (UID: \"9791465e-d449-49bf-8f13-7fb2253d53bf\") " pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp"
Jul 28 17:22:44.537006 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.536939    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:44.539263 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.539247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9791465e-d449-49bf-8f13-7fb2253d53bf-cert\") pod \"llmisvc-controller-manager-84fc959947-gw7kp\" (UID: \"9791465e-d449-49bf-8f13-7fb2253d53bf\") " pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp"
Jul 28 17:22:44.636453 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.636421    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:44.702236 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.702202    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:44.743746 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.743722    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp"
Jul 28 17:22:44.885891 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:44.885860    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-gw7kp"]
Jul 28 17:22:44.889731 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:22:44.889699    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9791465e_d449_49bf_8f13_7fb2253d53bf.slice/crio-a1f5feaeaed23acbb8b3f5edaa84c1c488b2e27456cc8ef6220a96b9443d2396 WatchSource:0}: Error finding container a1f5feaeaed23acbb8b3f5edaa84c1c488b2e27456cc8ef6220a96b9443d2396: Status 404 returned error can't find the container with id a1f5feaeaed23acbb8b3f5edaa84c1c488b2e27456cc8ef6220a96b9443d2396
Jul 28 17:22:45.023374 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:45.023330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp" event={"ID":"9791465e-d449-49bf-8f13-7fb2253d53bf","Type":"ContainerStarted","Data":"a1f5feaeaed23acbb8b3f5edaa84c1c488b2e27456cc8ef6220a96b9443d2396"}
Jul 28 17:22:45.024574 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:45.024547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq" event={"ID":"994c5f04-9e59-482e-8094-9e47898d2ca1","Type":"ContainerStarted","Data":"b2233ffecc35e570d349816c6c6cce574de7b43ccd636aaa4959b68ffde2cc5e"}
Jul 28 17:22:48.046680 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:48.046644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq" event={"ID":"994c5f04-9e59-482e-8094-9e47898d2ca1","Type":"ContainerStarted","Data":"1dc66b763615a1724dd6d47a389bd3ea6c51446ec2c6e40aa50ed1df10d9501e"}
Jul 28 17:22:48.047157 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:48.046767    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:22:48.064388 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:48.064337    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq" podStartSLOduration=2.08906592 podStartE2EDuration="5.064316004s" totalImagesPullingTime="2.975250084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:48.062389163 +0000 UTC m=+334.075769057" watchObservedRunningTime="2026-07-28 17:22:48.064316004 +0000 UTC m=+334.077695897"
Jul 28 17:22:50.056671 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:50.056639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp" event={"ID":"9791465e-d449-49bf-8f13-7fb2253d53bf","Type":"ContainerStarted","Data":"754258dff8a76cc7805bee52f36746aa63f1d909cc15bdbece5825f8c0edc419"}
Jul 28 17:22:50.057058 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:50.056776    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp"
Jul 28 17:22:50.073645 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:22:50.073591    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp" podStartSLOduration=2.8702594980000002 podStartE2EDuration="7.073573909s" totalImagesPullingTime="4.203314411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:50.072756153 +0000 UTC m=+336.086136048" watchObservedRunningTime="2026-07-28 17:22:50.073573909 +0000 UTC m=+336.086953843"
Jul 28 17:23:19.057294 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:19.057258    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:23:21.063268 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:21.063237    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-gw7kp"
Jul 28 17:23:30.101134 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.101100    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"]
Jul 28 17:23:30.101517 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.101310    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq" podUID="994c5f04-9e59-482e-8094-9e47898d2ca1" containerName="manager" containerID="cri-o://1dc66b763615a1724dd6d47a389bd3ea6c51446ec2c6e40aa50ed1df10d9501e" gracePeriod=10
Jul 28 17:23:30.126849 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.126822    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"]
Jul 28 17:23:30.129085 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.129066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"
Jul 28 17:23:30.143332 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.143298    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"]
Jul 28 17:23:30.215123 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.215085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztqn\" (UniqueName: \"kubernetes.io/projected/411c1fba-ee82-4396-9522-b44fa9ccfe41-kube-api-access-6ztqn\") pod \"kserve-controller-manager-85d8b8ddfc-ftbqg\" (UID: \"411c1fba-ee82-4396-9522-b44fa9ccfe41\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"
Jul 28 17:23:30.215297 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.215182    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/411c1fba-ee82-4396-9522-b44fa9ccfe41-cert\") pod \"kserve-controller-manager-85d8b8ddfc-ftbqg\" (UID: \"411c1fba-ee82-4396-9522-b44fa9ccfe41\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"
Jul 28 17:23:30.232039 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.231998    2583 generic.go:356] "Generic (PLEG): container finished" podID="994c5f04-9e59-482e-8094-9e47898d2ca1" containerID="1dc66b763615a1724dd6d47a389bd3ea6c51446ec2c6e40aa50ed1df10d9501e" exitCode=0
Jul 28 17:23:30.232193 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.232053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq" event={"ID":"994c5f04-9e59-482e-8094-9e47898d2ca1","Type":"ContainerDied","Data":"1dc66b763615a1724dd6d47a389bd3ea6c51446ec2c6e40aa50ed1df10d9501e"}
Jul 28 17:23:30.316118 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.316079    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztqn\" (UniqueName: \"kubernetes.io/projected/411c1fba-ee82-4396-9522-b44fa9ccfe41-kube-api-access-6ztqn\") pod \"kserve-controller-manager-85d8b8ddfc-ftbqg\" (UID: \"411c1fba-ee82-4396-9522-b44fa9ccfe41\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"
Jul 28 17:23:30.316282 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.316163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/411c1fba-ee82-4396-9522-b44fa9ccfe41-cert\") pod \"kserve-controller-manager-85d8b8ddfc-ftbqg\" (UID: \"411c1fba-ee82-4396-9522-b44fa9ccfe41\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"
Jul 28 17:23:30.318608 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.318577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/411c1fba-ee82-4396-9522-b44fa9ccfe41-cert\") pod \"kserve-controller-manager-85d8b8ddfc-ftbqg\" (UID: \"411c1fba-ee82-4396-9522-b44fa9ccfe41\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"
Jul 28 17:23:30.326129 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.326099    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztqn\" (UniqueName: \"kubernetes.io/projected/411c1fba-ee82-4396-9522-b44fa9ccfe41-kube-api-access-6ztqn\") pod \"kserve-controller-manager-85d8b8ddfc-ftbqg\" (UID: \"411c1fba-ee82-4396-9522-b44fa9ccfe41\") " pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"
Jul 28 17:23:30.344481 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.344456    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:23:30.416807 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.416729    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/994c5f04-9e59-482e-8094-9e47898d2ca1-cert\") pod \"994c5f04-9e59-482e-8094-9e47898d2ca1\" (UID: \"994c5f04-9e59-482e-8094-9e47898d2ca1\") "
Jul 28 17:23:30.416807 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.416775    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlmqx\" (UniqueName: \"kubernetes.io/projected/994c5f04-9e59-482e-8094-9e47898d2ca1-kube-api-access-wlmqx\") pod \"994c5f04-9e59-482e-8094-9e47898d2ca1\" (UID: \"994c5f04-9e59-482e-8094-9e47898d2ca1\") "
Jul 28 17:23:30.418934 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.418901    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994c5f04-9e59-482e-8094-9e47898d2ca1-kube-api-access-wlmqx" (OuterVolumeSpecName: "kube-api-access-wlmqx") pod "994c5f04-9e59-482e-8094-9e47898d2ca1" (UID: "994c5f04-9e59-482e-8094-9e47898d2ca1"). InnerVolumeSpecName "kube-api-access-wlmqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:23:30.418934 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.418923    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994c5f04-9e59-482e-8094-9e47898d2ca1-cert" (OuterVolumeSpecName: "cert") pod "994c5f04-9e59-482e-8094-9e47898d2ca1" (UID: "994c5f04-9e59-482e-8094-9e47898d2ca1"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:23:30.484406 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.484367    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"
Jul 28 17:23:30.517738 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.517697    2583 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/994c5f04-9e59-482e-8094-9e47898d2ca1-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:23:30.517738 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.517725    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlmqx\" (UniqueName: \"kubernetes.io/projected/994c5f04-9e59-482e-8094-9e47898d2ca1-kube-api-access-wlmqx\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:23:30.615868 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:30.615712    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"]
Jul 28 17:23:30.618370 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:23:30.618345    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411c1fba_ee82_4396_9522_b44fa9ccfe41.slice/crio-fb59cbe945b47fbc06dfc3c6aa8da4f1a412b178697f2d97e84121aa717a03c5 WatchSource:0}: Error finding container fb59cbe945b47fbc06dfc3c6aa8da4f1a412b178697f2d97e84121aa717a03c5: Status 404 returned error can't find the container with id fb59cbe945b47fbc06dfc3c6aa8da4f1a412b178697f2d97e84121aa717a03c5
Jul 28 17:23:31.238093 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:31.238060    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"
Jul 28 17:23:31.238093 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:31.238073    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-4rjpq" event={"ID":"994c5f04-9e59-482e-8094-9e47898d2ca1","Type":"ContainerDied","Data":"b2233ffecc35e570d349816c6c6cce574de7b43ccd636aaa4959b68ffde2cc5e"}
Jul 28 17:23:31.238580 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:31.238146    2583 scope.go:117] "RemoveContainer" containerID="1dc66b763615a1724dd6d47a389bd3ea6c51446ec2c6e40aa50ed1df10d9501e"
Jul 28 17:23:31.240135 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:31.239992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg" event={"ID":"411c1fba-ee82-4396-9522-b44fa9ccfe41","Type":"ContainerStarted","Data":"9ca828cd185ca9600d87423c740e565da51819494601895e0fe81dc3378f18a3"}
Jul 28 17:23:31.240135 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:31.240065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg" event={"ID":"411c1fba-ee82-4396-9522-b44fa9ccfe41","Type":"ContainerStarted","Data":"fb59cbe945b47fbc06dfc3c6aa8da4f1a412b178697f2d97e84121aa717a03c5"}
Jul 28 17:23:31.240135 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:31.240109    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"
Jul 28 17:23:31.259919 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:31.259872    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg" podStartSLOduration=0.779796576 podStartE2EDuration="1.259855714s" totalImagesPullingTime="480.059138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:31.25935123 +0000 UTC m=+377.272731124" watchObservedRunningTime="2026-07-28 17:23:31.259855714 +0000 UTC m=+377.273235636"
Jul 28 17:23:31.273189 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:31.273162    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"]
Jul 28 17:23:31.280139 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:31.280110    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-85d8b8ddfc-4rjpq"]
Jul 28 17:23:32.536712 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:32.536674    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="994c5f04-9e59-482e-8094-9e47898d2ca1" path="/var/lib/kubelet/pods/994c5f04-9e59-482e-8094-9e47898d2ca1/volumes"
Jul 28 17:23:41.973084 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:41.973039    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:52.935006 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:52.934969    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:53.036308 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:53.036277    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:53.136523 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:53.136484    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:55.992358 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:55.992324    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:56.063154 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:56.063123    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:57.736838 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:57.736802    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:57.807119 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:23:57.807076    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:02.250919 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:24:02.250893    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85d8b8ddfc-ftbqg"
Jul 28 17:24:03.399272 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:24:03.399233    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:03.490254 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:24:03.490206    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:03.591762 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:24:03.591728    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:04.276847 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:24:04.276806    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:04.348295 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:24:04.348264    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:45.874463 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:24:45.874370    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:59.712416 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:24:59.712381    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:59.816555 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:24:59.816522    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:44.955621 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:25:44.955580    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:08.921083 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:08.921050    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:08.923235 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:08.923212    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"]
Jul 28 17:26:08.923542 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:08.923530    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994c5f04-9e59-482e-8094-9e47898d2ca1" containerName="manager"
Jul 28 17:26:08.923588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:08.923544    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="994c5f04-9e59-482e-8094-9e47898d2ca1" containerName="manager"
Jul 28 17:26:08.923628 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:08.923599    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="994c5f04-9e59-482e-8094-9e47898d2ca1" containerName="manager"
Jul 28 17:26:08.925595 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:08.925581    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"
Jul 28 17:26:08.928005 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:08.927983    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 28 17:26:08.929068 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:08.929052    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 28 17:26:08.929154 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:08.929113    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9crzv\""
Jul 28 17:26:08.934619 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:08.934586    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"]
Jul 28 17:26:09.020449 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.020415    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:09.044387 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.044350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jf2\" (UniqueName: \"kubernetes.io/projected/dbc5dc0c-ae5d-4108-96e3-6d131274aa50-kube-api-access-c6jf2\") pod \"seaweedfs-tls-serving-67b7477c6d-6k2nm\" (UID: \"dbc5dc0c-ae5d-4108-96e3-6d131274aa50\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"
Jul 28 17:26:09.044517 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.044397    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/dbc5dc0c-ae5d-4108-96e3-6d131274aa50-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-6k2nm\" (UID: \"dbc5dc0c-ae5d-4108-96e3-6d131274aa50\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"
Jul 28 17:26:09.044517 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.044425    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dbc5dc0c-ae5d-4108-96e3-6d131274aa50-data\") pod \"seaweedfs-tls-serving-67b7477c6d-6k2nm\" (UID: \"dbc5dc0c-ae5d-4108-96e3-6d131274aa50\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"
Jul 28 17:26:09.145735 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.145693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dbc5dc0c-ae5d-4108-96e3-6d131274aa50-data\") pod \"seaweedfs-tls-serving-67b7477c6d-6k2nm\" (UID: \"dbc5dc0c-ae5d-4108-96e3-6d131274aa50\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"
Jul 28 17:26:09.145904 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.145771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jf2\" (UniqueName: \"kubernetes.io/projected/dbc5dc0c-ae5d-4108-96e3-6d131274aa50-kube-api-access-c6jf2\") pod \"seaweedfs-tls-serving-67b7477c6d-6k2nm\" (UID: \"dbc5dc0c-ae5d-4108-96e3-6d131274aa50\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"
Jul 28 17:26:09.145904 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.145805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/dbc5dc0c-ae5d-4108-96e3-6d131274aa50-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-6k2nm\" (UID: \"dbc5dc0c-ae5d-4108-96e3-6d131274aa50\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"
Jul 28 17:26:09.146141 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.146125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dbc5dc0c-ae5d-4108-96e3-6d131274aa50-data\") pod \"seaweedfs-tls-serving-67b7477c6d-6k2nm\" (UID: \"dbc5dc0c-ae5d-4108-96e3-6d131274aa50\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"
Jul 28 17:26:09.148228 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.148199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/dbc5dc0c-ae5d-4108-96e3-6d131274aa50-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-6k2nm\" (UID: \"dbc5dc0c-ae5d-4108-96e3-6d131274aa50\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"
Jul 28 17:26:09.154662 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.154637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jf2\" (UniqueName: \"kubernetes.io/projected/dbc5dc0c-ae5d-4108-96e3-6d131274aa50-kube-api-access-c6jf2\") pod \"seaweedfs-tls-serving-67b7477c6d-6k2nm\" (UID: \"dbc5dc0c-ae5d-4108-96e3-6d131274aa50\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"
Jul 28 17:26:09.234807 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.234764    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"
Jul 28 17:26:09.372044 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.372001    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm"]
Jul 28 17:26:09.374685 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:26:09.374655    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc5dc0c_ae5d_4108_96e3_6d131274aa50.slice/crio-b088431438244ef1d24eb592c284370c925c8a024e5f0ea4ada4ba9f68026a24 WatchSource:0}: Error finding container b088431438244ef1d24eb592c284370c925c8a024e5f0ea4ada4ba9f68026a24: Status 404 returned error can't find the container with id b088431438244ef1d24eb592c284370c925c8a024e5f0ea4ada4ba9f68026a24
Jul 28 17:26:09.893163 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:09.893091    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm" event={"ID":"dbc5dc0c-ae5d-4108-96e3-6d131274aa50","Type":"ContainerStarted","Data":"b088431438244ef1d24eb592c284370c925c8a024e5f0ea4ada4ba9f68026a24"}
Jul 28 17:26:12.907738 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:12.907701    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm" event={"ID":"dbc5dc0c-ae5d-4108-96e3-6d131274aa50","Type":"ContainerStarted","Data":"2d65536ce6f450f9366cb36971b0d9e6e70e4d224675ebef2254ede482ab68ea"}
Jul 28 17:26:12.926998 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:12.926925    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-6k2nm" podStartSLOduration=1.892273557 podStartE2EDuration="4.926904251s" totalImagesPullingTime="3.034630694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:12.92461728 +0000 UTC m=+538.937997171" watchObservedRunningTime="2026-07-28 17:26:12.926904251 +0000 UTC m=+538.940284148"
Jul 28 17:26:29.718793 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:29.718756    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:30.781728 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:30.781678    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:30.881820 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:30.881783    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:42.606555 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:42.606522    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:42.783213 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:26:42.783178    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:14.392962 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:27:14.392925    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:27:14.398950 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:27:14.398932    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:27:58.552180 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:27:58.552145    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:59.917753 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:27:59.917718    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:12.260745 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:28:12.260707    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:27.826817 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:28:27.826784    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:27.912705 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:28:27.912669    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:27.983396 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:28:27.983360    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:03.064286 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:30:03.064241    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:03.131661 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:30:03.131623    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:03.188097 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:30:03.188068    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:09.439990 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:30:09.439954    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:23.102456 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:30:23.102414    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:23.447883 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:30:23.447800    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:39.717667 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:30:39.717568    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:58.393383 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:31:58.393344    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:58.466154 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:31:58.466118    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:58.535108 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:31:58.535072    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:58.634894 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:31:58.634857    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:14.420211 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:32:14.420181    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:32:14.424560 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:32:14.424539    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:34:54.798864 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:34:54.798806    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:59.879381 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:34:59.879348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:00.183662 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:35:00.183575    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:18.558570 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:36:18.558534    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:20.146415 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:36:20.146384    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:56.147795 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:36:56.147756    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:01.220611 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:37:01.220575    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:01.483439 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:37:01.483404    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:14.448779 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:37:14.448749    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:37:14.450198 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:37:14.450181    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:37:27.086524 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:37:27.086491    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:58.153260 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:38:58.153223    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:02.953976 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:39:02.953937    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:03.450096 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:39:03.450056    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:49.314287 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:40:49.314247    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:54.398611 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:40:54.398566    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:54.736524 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:40:54.736491    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:37.620959 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:41:37.620913    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:42.051744 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:41:42.051709    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:42.122148 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:41:42.122112    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:56.476091 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:41:56.476053    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:58.875043 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:41:58.874991    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:14.474825 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:42:14.474794    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:42:14.475374 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:42:14.474885    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:42:17.341057 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:42:17.341006    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:22.486926 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:42:22.486885    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:22.556556 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:42:22.556526    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:22.775429 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:42:22.775345    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:23.322588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:42:23.322541    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:52.677350 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:43:52.677309    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:52.736471 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:43:52.736435    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:57.756214 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:43:57.756169    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:57.849469 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:43:57.849429    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:57.927580 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:43:57.927551    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:58.396752 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:43:58.396713    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:22.911427 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:22.911389    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:22.914992 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:22.914972    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"]
Jul 28 17:45:22.918346 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:22.918329    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:22.920940 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:22.920921    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 17:45:22.920940 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:22.920933    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:45:22.921217 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:22.921203    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:45:22.922085 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:22.922068    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 17:45:22.922193 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:22.922174    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:45:22.922264 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:22.922182    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:45:22.922264 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:22.922183    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:45:22.929418 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:22.929397    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"]
Jul 28 17:45:23.054873 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.054835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.055074 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.054881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2456333-68ac-4751-9415-7897c501d135-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.055074 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.054930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvv5\" (UniqueName: \"kubernetes.io/projected/b2456333-68ac-4751-9415-7897c501d135-kube-api-access-wsvv5\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.055074 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.054969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2456333-68ac-4751-9415-7897c501d135-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.055074 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.055006    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.155510 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.155472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.155689 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.155522    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.155689 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.155546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2456333-68ac-4751-9415-7897c501d135-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.155689 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.155567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvv5\" (UniqueName: \"kubernetes.io/projected/b2456333-68ac-4751-9415-7897c501d135-kube-api-access-wsvv5\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.155689 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.155596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2456333-68ac-4751-9415-7897c501d135-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.156007 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.155988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2456333-68ac-4751-9415-7897c501d135-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.156299 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.156279    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.156385 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.156283    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.158242 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.158219    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2456333-68ac-4751-9415-7897c501d135-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.164148 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.164097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvv5\" (UniqueName: \"kubernetes.io/projected/b2456333-68ac-4751-9415-7897c501d135-kube-api-access-wsvv5\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5mx8p\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.229079 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.229043    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:23.361823 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.361787    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"]
Jul 28 17:45:23.365330 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:45:23.365302    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2456333_68ac_4751_9415_7897c501d135.slice/crio-f475a7525c63eb1dd8074511eeb8e0949c4c1ce074467c0b65331544886ca248 WatchSource:0}: Error finding container f475a7525c63eb1dd8074511eeb8e0949c4c1ce074467c0b65331544886ca248: Status 404 returned error can't find the container with id f475a7525c63eb1dd8074511eeb8e0949c4c1ce074467c0b65331544886ca248
Jul 28 17:45:23.367196 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.367178    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:45:23.737535 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:23.737501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" event={"ID":"b2456333-68ac-4751-9415-7897c501d135","Type":"ContainerStarted","Data":"f475a7525c63eb1dd8074511eeb8e0949c4c1ce074467c0b65331544886ca248"}
Jul 28 17:45:27.758150 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:27.758117    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" event={"ID":"b2456333-68ac-4751-9415-7897c501d135","Type":"ContainerStarted","Data":"ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32"}
Jul 28 17:45:30.771405 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:30.771367    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2456333-68ac-4751-9415-7897c501d135" containerID="ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32" exitCode=0
Jul 28 17:45:30.771765 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:30.771417    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" event={"ID":"b2456333-68ac-4751-9415-7897c501d135","Type":"ContainerDied","Data":"ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32"}
Jul 28 17:45:44.851112 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:44.850993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" event={"ID":"b2456333-68ac-4751-9415-7897c501d135","Type":"ContainerStarted","Data":"622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec"}
Jul 28 17:45:47.868509 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:47.868471    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" event={"ID":"b2456333-68ac-4751-9415-7897c501d135","Type":"ContainerStarted","Data":"8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1"}
Jul 28 17:45:47.868895 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:47.868705    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:47.868895 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:47.868749    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:47.869876 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:47.869852    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:45:47.889414 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:47.889358    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podStartSLOduration=2.580902294 podStartE2EDuration="25.889342067s" totalImagesPullingTime="19.598134125s" totalInitContainerRuntime="3.710305648s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:47.887500319 +0000 UTC m=+1713.900880212" watchObservedRunningTime="2026-07-28 17:45:47.889342067 +0000 UTC m=+1713.902721960"
Jul 28 17:45:48.873151 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:48.873110    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:45:53.878198 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:53.878170    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:45:53.878707 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:45:53.878679    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:46:00.573204 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:46:00.573172    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:03.879529 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:46:03.879491    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:46:13.879196 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:46:13.879153    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:46:23.878867 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:46:23.878809    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:46:33.879441 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:46:33.879398    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:46:43.879600 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:46:43.879560    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:46:53.879483 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:46:53.879445    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:47:03.879890 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:03.879861    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:47:13.168702 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:13.168664    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"]
Jul 28 17:47:13.169177 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:13.168904    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container" containerID="cri-o://622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec" gracePeriod=30
Jul 28 17:47:13.169177 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:13.168939    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kube-rbac-proxy" containerID="cri-o://8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1" gracePeriod=30
Jul 28 17:47:13.197156 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:13.197113    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:13.874102 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:13.874060    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 17:47:13.878709 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:13.878678    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:47:14.245056 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:14.245004    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2456333-68ac-4751-9415-7897c501d135" containerID="8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1" exitCode=2
Jul 28 17:47:14.245461 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:14.245077    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" event={"ID":"b2456333-68ac-4751-9415-7897c501d135","Type":"ContainerDied","Data":"8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1"}
Jul 28 17:47:14.501010 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:14.500925    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:47:14.501674 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:14.501655    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:47:18.011165 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.011137    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:47:18.123146 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.123070    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2456333-68ac-4751-9415-7897c501d135-proxy-tls\") pod \"b2456333-68ac-4751-9415-7897c501d135\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") "
Jul 28 17:47:18.123146 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.123112    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsvv5\" (UniqueName: \"kubernetes.io/projected/b2456333-68ac-4751-9415-7897c501d135-kube-api-access-wsvv5\") pod \"b2456333-68ac-4751-9415-7897c501d135\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") "
Jul 28 17:47:18.123146 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.123139    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2456333-68ac-4751-9415-7897c501d135-kserve-provision-location\") pod \"b2456333-68ac-4751-9415-7897c501d135\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") "
Jul 28 17:47:18.123660 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.123182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-kube-rbac-proxy-sar-config\") pod \"b2456333-68ac-4751-9415-7897c501d135\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") "
Jul 28 17:47:18.123660 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.123231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-cabundle-cert\") pod \"b2456333-68ac-4751-9415-7897c501d135\" (UID: \"b2456333-68ac-4751-9415-7897c501d135\") "
Jul 28 17:47:18.123660 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.123567    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2456333-68ac-4751-9415-7897c501d135-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2456333-68ac-4751-9415-7897c501d135" (UID: "b2456333-68ac-4751-9415-7897c501d135"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:18.123812 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.123706    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2456333-68ac-4751-9415-7897c501d135" (UID: "b2456333-68ac-4751-9415-7897c501d135"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:47:18.123812 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.123735    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2456333-68ac-4751-9415-7897c501d135" (UID: "b2456333-68ac-4751-9415-7897c501d135"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:47:18.125518 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.125488    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2456333-68ac-4751-9415-7897c501d135-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2456333-68ac-4751-9415-7897c501d135" (UID: "b2456333-68ac-4751-9415-7897c501d135"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:18.125633 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.125525    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2456333-68ac-4751-9415-7897c501d135-kube-api-access-wsvv5" (OuterVolumeSpecName: "kube-api-access-wsvv5") pod "b2456333-68ac-4751-9415-7897c501d135" (UID: "b2456333-68ac-4751-9415-7897c501d135"). InnerVolumeSpecName "kube-api-access-wsvv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:18.223939 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.223902    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2456333-68ac-4751-9415-7897c501d135-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:47:18.223939 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.223933    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsvv5\" (UniqueName: \"kubernetes.io/projected/b2456333-68ac-4751-9415-7897c501d135-kube-api-access-wsvv5\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:47:18.223939 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.223945    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2456333-68ac-4751-9415-7897c501d135-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:47:18.224205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.223959    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:47:18.224205 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.223972    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2456333-68ac-4751-9415-7897c501d135-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:47:18.263555 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.263521    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2456333-68ac-4751-9415-7897c501d135" containerID="622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec" exitCode=0
Jul 28 17:47:18.263743 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.263588    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"
Jul 28 17:47:18.263743 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.263594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" event={"ID":"b2456333-68ac-4751-9415-7897c501d135","Type":"ContainerDied","Data":"622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec"}
Jul 28 17:47:18.263743 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.263653    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p" event={"ID":"b2456333-68ac-4751-9415-7897c501d135","Type":"ContainerDied","Data":"f475a7525c63eb1dd8074511eeb8e0949c4c1ce074467c0b65331544886ca248"}
Jul 28 17:47:18.263743 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.263679    2583 scope.go:117] "RemoveContainer" containerID="8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1"
Jul 28 17:47:18.273417 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.273374    2583 scope.go:117] "RemoveContainer" containerID="622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec"
Jul 28 17:47:18.290163 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.290130    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"]
Jul 28 17:47:18.291755 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.291728    2583 scope.go:117] "RemoveContainer" containerID="ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32"
Jul 28 17:47:18.293753 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.293729    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p"]
Jul 28 17:47:18.299966 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.299933    2583 scope.go:117] "RemoveContainer" containerID="8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1"
Jul 28 17:47:18.300526 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:47:18.300478    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1\": container with ID starting with 8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1 not found: ID does not exist" containerID="8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1"
Jul 28 17:47:18.300602 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.300529    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1"} err="failed to get container status \"8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1\": rpc error: code = NotFound desc = could not find container \"8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1\": container with ID starting with 8f509dda4b8d8da224a6af99ee493ad7253f928ff5106b7492fa84e3282024a1 not found: ID does not exist"
Jul 28 17:47:18.300602 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.300589    2583 scope.go:117] "RemoveContainer" containerID="622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec"
Jul 28 17:47:18.300904 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:47:18.300882    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec\": container with ID starting with 622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec not found: ID does not exist" containerID="622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec"
Jul 28 17:47:18.300996 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.300906    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec"} err="failed to get container status \"622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec\": rpc error: code = NotFound desc = could not find container \"622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec\": container with ID starting with 622664d947bca51bfc1f05fd7d2a1bd861e4005a4d73042955668ccf6519a1ec not found: ID does not exist"
Jul 28 17:47:18.300996 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.300921    2583 scope.go:117] "RemoveContainer" containerID="ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32"
Jul 28 17:47:18.301177 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:47:18.301153    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32\": container with ID starting with ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32 not found: ID does not exist" containerID="ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32"
Jul 28 17:47:18.301225 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.301176    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32"} err="failed to get container status \"ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32\": rpc error: code = NotFound desc = could not find container \"ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32\": container with ID starting with ec06cce1399041f761095410d922e0e0dcda4b3180e7fac728f078fc393fcf32 not found: ID does not exist"
Jul 28 17:47:18.313405 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.313372    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:18.360853 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.360821    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"]
Jul 28 17:47:18.361254 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.361240    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kube-rbac-proxy"
Jul 28 17:47:18.361304 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.361255    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kube-rbac-proxy"
Jul 28 17:47:18.361304 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.361267    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container"
Jul 28 17:47:18.361304 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.361272    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container"
Jul 28 17:47:18.361304 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.361287    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="storage-initializer"
Jul 28 17:47:18.361304 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.361296    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="storage-initializer"
Jul 28 17:47:18.361455 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.361351    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kube-rbac-proxy"
Jul 28 17:47:18.361455 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.361360    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2456333-68ac-4751-9415-7897c501d135" containerName="kserve-container"
Jul 28 17:47:18.365595 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.365577    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.368636 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.368614    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:47:18.368987 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.368868    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 17:47:18.368987 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.368919    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:47:18.368987 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.368928    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 17:47:18.368987 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.368952    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:47:18.369238 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.368952    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:47:18.369238 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.368928    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:47:18.378849 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.378796    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"]
Jul 28 17:47:18.399177 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.399143    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:18.425599 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.425565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.425744 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.425616    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.425744 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.425646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsmn7\" (UniqueName: \"kubernetes.io/projected/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-api-access-zsmn7\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.425744 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.425692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.425744 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.425737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.478252 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.478213    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:18.527147 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.527104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.527416 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.527161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsmn7\" (UniqueName: \"kubernetes.io/projected/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-api-access-zsmn7\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.527416 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.527241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.527416 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.527280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.527416 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.527331    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.527732 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.527709    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.527792 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.527762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.527922 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.527907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.529848 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.529820    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.537636 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.537606    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsmn7\" (UniqueName: \"kubernetes.io/projected/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-api-access-zsmn7\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.538371 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.538349    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2456333-68ac-4751-9415-7897c501d135" path="/var/lib/kubelet/pods/b2456333-68ac-4751-9415-7897c501d135/volumes"
Jul 28 17:47:18.581336 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.581298    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:18.676675 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.676581    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:18.810256 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:18.810229    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"]
Jul 28 17:47:18.815248 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:47:18.815192    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b71f8a_ea80_4c7e_8b54_060e5ecc797d.slice/crio-dd9ee07f2fbcf79f59af17ea96d44fac6a9cd6684246a0e2ec17b23d7b2931ad WatchSource:0}: Error finding container dd9ee07f2fbcf79f59af17ea96d44fac6a9cd6684246a0e2ec17b23d7b2931ad: Status 404 returned error can't find the container with id dd9ee07f2fbcf79f59af17ea96d44fac6a9cd6684246a0e2ec17b23d7b2931ad
Jul 28 17:47:19.270336 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:19.270292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" event={"ID":"94b71f8a-ea80-4c7e-8b54-060e5ecc797d","Type":"ContainerStarted","Data":"845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339"}
Jul 28 17:47:19.270788 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:19.270368    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" event={"ID":"94b71f8a-ea80-4c7e-8b54-060e5ecc797d","Type":"ContainerStarted","Data":"dd9ee07f2fbcf79f59af17ea96d44fac6a9cd6684246a0e2ec17b23d7b2931ad"}
Jul 28 17:47:23.289347 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:23.289311    2583 generic.go:356] "Generic (PLEG): container finished" podID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerID="845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339" exitCode=0
Jul 28 17:47:23.289782 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:23.289384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" event={"ID":"94b71f8a-ea80-4c7e-8b54-060e5ecc797d","Type":"ContainerDied","Data":"845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339"}
Jul 28 17:47:24.296158 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:24.296123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" event={"ID":"94b71f8a-ea80-4c7e-8b54-060e5ecc797d","Type":"ContainerStarted","Data":"0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997"}
Jul 28 17:47:24.296524 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:24.296185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" event={"ID":"94b71f8a-ea80-4c7e-8b54-060e5ecc797d","Type":"ContainerStarted","Data":"523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378"}
Jul 28 17:47:24.296613 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:24.296545    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:24.296700 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:24.296681    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:24.298081 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:24.298058    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:47:24.316070 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:24.316006    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podStartSLOduration=2.594033753 podStartE2EDuration="6.315988735s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.721954982s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:24.314898375 +0000 UTC m=+1810.328278293" watchObservedRunningTime="2026-07-28 17:47:24.315988735 +0000 UTC m=+1810.329368628"
Jul 28 17:47:25.300897 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:25.300857    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:47:26.305789 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:26.305746    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:47:31.311634 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:31.311604    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:47:31.312144 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:31.312117    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:47:41.312525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:41.312483    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:47:51.312266 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:47:51.312219    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:48:01.312069 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:48:01.312031    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:48:11.312485 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:48:11.312447    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:48:21.312482 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:48:21.312441    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:48:31.312673 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:48:31.312634    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:48:35.458492 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:48:35.458460    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:37.533785 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:48:37.533748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:48:43.367448 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:48:43.367410    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:13.561420 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:13.561387    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"]
Jul 28 17:50:13.561897 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:13.561630    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" containerID="cri-o://523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378" gracePeriod=30
Jul 28 17:50:13.561897 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:13.561682    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kube-rbac-proxy" containerID="cri-o://0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997" gracePeriod=30
Jul 28 17:50:13.621256 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:13.621224    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:14.018362 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:14.018329    2583 generic.go:356] "Generic (PLEG): container finished" podID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerID="0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997" exitCode=2
Jul 28 17:50:14.018525 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:14.018401    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" event={"ID":"94b71f8a-ea80-4c7e-8b54-060e5ecc797d","Type":"ContainerDied","Data":"0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997"}
Jul 28 17:50:14.833416 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:14.833385    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:16.306445 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:16.306399    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 17:50:17.532753 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:17.532709    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:50:18.404459 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.404435    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:50:18.467313 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.467278    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-rbac-proxy-sar-config\") pod \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") "
Jul 28 17:50:18.467490 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.467325    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsmn7\" (UniqueName: \"kubernetes.io/projected/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-api-access-zsmn7\") pod \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") "
Jul 28 17:50:18.467490 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.467365    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kserve-provision-location\") pod \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") "
Jul 28 17:50:18.467490 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.467477    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-proxy-tls\") pod \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") "
Jul 28 17:50:18.467615 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.467522    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-cabundle-cert\") pod \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\" (UID: \"94b71f8a-ea80-4c7e-8b54-060e5ecc797d\") "
Jul 28 17:50:18.467696 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.467671    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94b71f8a-ea80-4c7e-8b54-060e5ecc797d" (UID: "94b71f8a-ea80-4c7e-8b54-060e5ecc797d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:18.467759 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.467710    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94b71f8a-ea80-4c7e-8b54-060e5ecc797d" (UID: "94b71f8a-ea80-4c7e-8b54-060e5ecc797d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:50:18.467821 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.467768    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:50:18.467894 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.467879    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:50:18.467969 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.467945    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "94b71f8a-ea80-4c7e-8b54-060e5ecc797d" (UID: "94b71f8a-ea80-4c7e-8b54-060e5ecc797d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:50:18.469566 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.469537    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-api-access-zsmn7" (OuterVolumeSpecName: "kube-api-access-zsmn7") pod "94b71f8a-ea80-4c7e-8b54-060e5ecc797d" (UID: "94b71f8a-ea80-4c7e-8b54-060e5ecc797d"). InnerVolumeSpecName "kube-api-access-zsmn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:50:18.469669 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.469622    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94b71f8a-ea80-4c7e-8b54-060e5ecc797d" (UID: "94b71f8a-ea80-4c7e-8b54-060e5ecc797d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:50:18.568401 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.568369    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsmn7\" (UniqueName: \"kubernetes.io/projected/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-kube-api-access-zsmn7\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:50:18.568401 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.568394    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:50:18.568401 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.568404    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94b71f8a-ea80-4c7e-8b54-060e5ecc797d-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:50:18.655172 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.655123    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:18.738544 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.738510    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:18.817705 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.817630    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:18.917350 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:18.917322    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:19.042725 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.042690    2583 generic.go:356] "Generic (PLEG): container finished" podID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerID="523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378" exitCode=0
Jul 28 17:50:19.042884 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.042742    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"
Jul 28 17:50:19.042884 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.042749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" event={"ID":"94b71f8a-ea80-4c7e-8b54-060e5ecc797d","Type":"ContainerDied","Data":"523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378"}
Jul 28 17:50:19.042884 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.042806    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z" event={"ID":"94b71f8a-ea80-4c7e-8b54-060e5ecc797d","Type":"ContainerDied","Data":"dd9ee07f2fbcf79f59af17ea96d44fac6a9cd6684246a0e2ec17b23d7b2931ad"}
Jul 28 17:50:19.042884 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.042835    2583 scope.go:117] "RemoveContainer" containerID="0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997"
Jul 28 17:50:19.051102 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.051082    2583 scope.go:117] "RemoveContainer" containerID="523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378"
Jul 28 17:50:19.060425 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.060400    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"]
Jul 28 17:50:19.062002 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.061980    2583 scope.go:117] "RemoveContainer" containerID="845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339"
Jul 28 17:50:19.067285 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.067265    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z"]
Jul 28 17:50:19.070378 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.070359    2583 scope.go:117] "RemoveContainer" containerID="0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997"
Jul 28 17:50:19.070624 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:50:19.070606    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997\": container with ID starting with 0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997 not found: ID does not exist" containerID="0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997"
Jul 28 17:50:19.070668 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.070639    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997"} err="failed to get container status \"0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997\": rpc error: code = NotFound desc = could not find container \"0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997\": container with ID starting with 0c4df1b236985d2962374fa88487561141e4f5c5255769d3fb82e43edb0ad997 not found: ID does not exist"
Jul 28 17:50:19.070668 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.070661    2583 scope.go:117] "RemoveContainer" containerID="523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378"
Jul 28 17:50:19.070876 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:50:19.070859    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378\": container with ID starting with 523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378 not found: ID does not exist" containerID="523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378"
Jul 28 17:50:19.070914 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.070883    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378"} err="failed to get container status \"523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378\": rpc error: code = NotFound desc = could not find container \"523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378\": container with ID starting with 523b85ad43fcabb4c2544383d378268c08281527d7ead79f2268e2a92ae44378 not found: ID does not exist"
Jul 28 17:50:19.070914 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.070898    2583 scope.go:117] "RemoveContainer" containerID="845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339"
Jul 28 17:50:19.071132 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:50:19.071117    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339\": container with ID starting with 845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339 not found: ID does not exist" containerID="845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339"
Jul 28 17:50:19.071180 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:19.071136    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339"} err="failed to get container status \"845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339\": rpc error: code = NotFound desc = could not find container \"845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339\": container with ID starting with 845d2d8333f7bc2b3d926c6a39a805b1c6f0c01a59c66d38afc8d7cedcbb7339 not found: ID does not exist"
Jul 28 17:50:20.543815 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:50:20.543781    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" path="/var/lib/kubelet/pods/94b71f8a-ea80-4c7e-8b54-060e5ecc797d/volumes"
Jul 28 17:51:37.520587 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:37.520509    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:48.768539 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:48.768493    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:48.834295 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:48.834251    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:59.174360 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.174327    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:59.293207 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.293171    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"]
Jul 28 17:51:59.293583 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.293568    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="storage-initializer"
Jul 28 17:51:59.293630 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.293586    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="storage-initializer"
Jul 28 17:51:59.293630 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.293598    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kube-rbac-proxy"
Jul 28 17:51:59.293630 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.293604    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kube-rbac-proxy"
Jul 28 17:51:59.293630 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.293615    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container"
Jul 28 17:51:59.293630 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.293622    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container"
Jul 28 17:51:59.293778 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.293672    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kube-rbac-proxy"
Jul 28 17:51:59.293778 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.293681    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b71f8a-ea80-4c7e-8b54-060e5ecc797d" containerName="kserve-container"
Jul 28 17:51:59.296852 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.296832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.303784 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.303765    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 28 17:51:59.304059 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.304044    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:51:59.304146 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.304056    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 17:51:59.304872 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.304858    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:51:59.304938 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.304895    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:51:59.304938 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.304926    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:51:59.305064 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.304940    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:51:59.314808 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.314788    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"]
Jul 28 17:51:59.392771 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.392741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.392966 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.392786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.392966 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.392807    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.392966 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.392882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55ng9\" (UniqueName: \"kubernetes.io/projected/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-api-access-55ng9\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.392966 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.392929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.417911 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.417876    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:59.494026 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.493998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.494224 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.494062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.494224 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.494083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.494224 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.494110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ng9\" (UniqueName: \"kubernetes.io/projected/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-api-access-55ng9\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.494224 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.494133    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.494588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.494568    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.494847 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.494824    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.494896 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.494857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.496571 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.496550    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.504209 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.504189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ng9\" (UniqueName: \"kubernetes.io/projected/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-api-access-55ng9\") pod \"isvc-lightgbm-predictor-54f85ddf88-cx5mp\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.606784 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.606744    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:51:59.748934 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:51:59.748856    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"]
Jul 28 17:51:59.753514 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:51:59.753485    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa9cd9a_4a1d_4150_bc93_303ff34c6c63.slice/crio-8120d23e9791d17bb7e9454428ba300e1bd5956f8284151a321b56337b95bdcb WatchSource:0}: Error finding container 8120d23e9791d17bb7e9454428ba300e1bd5956f8284151a321b56337b95bdcb: Status 404 returned error can't find the container with id 8120d23e9791d17bb7e9454428ba300e1bd5956f8284151a321b56337b95bdcb
Jul 28 17:52:00.469524 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:00.469487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" event={"ID":"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63","Type":"ContainerStarted","Data":"656c044bd9c6c3b3cb02a520460479ae1c597b7cea2d8084c2a696dd8e52e996"}
Jul 28 17:52:00.469917 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:00.469545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" event={"ID":"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63","Type":"ContainerStarted","Data":"8120d23e9791d17bb7e9454428ba300e1bd5956f8284151a321b56337b95bdcb"}
Jul 28 17:52:03.484283 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:03.484243    2583 generic.go:356] "Generic (PLEG): container finished" podID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerID="656c044bd9c6c3b3cb02a520460479ae1c597b7cea2d8084c2a696dd8e52e996" exitCode=0
Jul 28 17:52:03.484638 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:03.484312    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" event={"ID":"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63","Type":"ContainerDied","Data":"656c044bd9c6c3b3cb02a520460479ae1c597b7cea2d8084c2a696dd8e52e996"}
Jul 28 17:52:03.485681 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:03.485665    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:52:10.521978 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:10.521939    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" event={"ID":"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63","Type":"ContainerStarted","Data":"a22f2fba14dca9bb0181de596a86c0126e09d7d7dd8e738a2c365ac62404eb02"}
Jul 28 17:52:10.522361 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:10.522044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" event={"ID":"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63","Type":"ContainerStarted","Data":"3b507f865e1c8bd6a8a20d4abc79dfcc5bb0d1595e344baffa275455feda5b3e"}
Jul 28 17:52:10.522408 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:10.522356    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:52:10.522502 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:10.522484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:52:10.524002 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:10.523975    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:52:10.546443 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:10.546399    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podStartSLOduration=1.3895778509999999 podStartE2EDuration="11.546384855s" totalImagesPullingTime="6.530363022s" totalInitContainerRuntime="3.626443982s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:10.545442374 +0000 UTC m=+2096.558822268" watchObservedRunningTime="2026-07-28 17:52:10.546384855 +0000 UTC m=+2096.559764748"
Jul 28 17:52:11.526376 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:11.526340    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:52:14.527416 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:14.527390    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:52:14.527918 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:14.527901    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:52:16.531194 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:16.531165    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:52:16.531699 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:16.531675    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:52:26.531788 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:26.531743    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:52:36.532284 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:36.532246    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:52:46.532091 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:46.532049    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:52:56.532138 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:52:56.532095    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:53:06.532498 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:06.532419    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:53:16.532653 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:16.532610    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:53:26.536647 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:26.536615    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:53:29.229830 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:29.229800    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"]
Jul 28 17:53:29.230259 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:29.229999    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container" containerID="cri-o://3b507f865e1c8bd6a8a20d4abc79dfcc5bb0d1595e344baffa275455feda5b3e" gracePeriod=30
Jul 28 17:53:29.230259 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:29.230057    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kube-rbac-proxy" containerID="cri-o://a22f2fba14dca9bb0181de596a86c0126e09d7d7dd8e738a2c365ac62404eb02" gracePeriod=30
Jul 28 17:53:29.265065 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:29.265035    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:29.858070 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:29.858031    2583 generic.go:356] "Generic (PLEG): container finished" podID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerID="a22f2fba14dca9bb0181de596a86c0126e09d7d7dd8e738a2c365ac62404eb02" exitCode=2
Jul 28 17:53:29.858246 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:29.858113    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" event={"ID":"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63","Type":"ContainerDied","Data":"a22f2fba14dca9bb0181de596a86c0126e09d7d7dd8e738a2c365ac62404eb02"}
Jul 28 17:53:31.526807 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:31.526761    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 17:53:33.876744 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.876716    2583 generic.go:356] "Generic (PLEG): container finished" podID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerID="3b507f865e1c8bd6a8a20d4abc79dfcc5bb0d1595e344baffa275455feda5b3e" exitCode=0
Jul 28 17:53:33.877056 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.876791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" event={"ID":"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63","Type":"ContainerDied","Data":"3b507f865e1c8bd6a8a20d4abc79dfcc5bb0d1595e344baffa275455feda5b3e"}
Jul 28 17:53:33.877056 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.876852    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:53:33.877056 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.876858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp" event={"ID":"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63","Type":"ContainerDied","Data":"8120d23e9791d17bb7e9454428ba300e1bd5956f8284151a321b56337b95bdcb"}
Jul 28 17:53:33.877056 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.876887    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8120d23e9791d17bb7e9454428ba300e1bd5956f8284151a321b56337b95bdcb"
Jul 28 17:53:33.999046 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.998986    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kserve-provision-location\") pod \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") "
Jul 28 17:53:33.999213 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.999062    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-cabundle-cert\") pod \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") "
Jul 28 17:53:33.999213 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.999102    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-proxy-tls\") pod \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") "
Jul 28 17:53:33.999213 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.999134    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55ng9\" (UniqueName: \"kubernetes.io/projected/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-api-access-55ng9\") pod \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") "
Jul 28 17:53:33.999213 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.999161    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-rbac-proxy-sar-config\") pod \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\" (UID: \"3aa9cd9a-4a1d-4150-bc93-303ff34c6c63\") "
Jul 28 17:53:33.999437 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.999363    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" (UID: "3aa9cd9a-4a1d-4150-bc93-303ff34c6c63"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:33.999567 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.999536    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" (UID: "3aa9cd9a-4a1d-4150-bc93-303ff34c6c63"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:53:33.999671 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:33.999576    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" (UID: "3aa9cd9a-4a1d-4150-bc93-303ff34c6c63"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:53:34.001334 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.001305    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-api-access-55ng9" (OuterVolumeSpecName: "kube-api-access-55ng9") pod "3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" (UID: "3aa9cd9a-4a1d-4150-bc93-303ff34c6c63"). InnerVolumeSpecName "kube-api-access-55ng9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:34.001421 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.001365    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" (UID: "3aa9cd9a-4a1d-4150-bc93-303ff34c6c63"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:34.100438 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.100400    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55ng9\" (UniqueName: \"kubernetes.io/projected/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-api-access-55ng9\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:53:34.100438 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.100428    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:53:34.100438 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.100437    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:53:34.100661 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.100450    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:53:34.100661 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.100459    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:53:34.331654 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.331577    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:34.635365 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.635284    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:34.881030 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.880999    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"
Jul 28 17:53:34.901036 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.900935    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"]
Jul 28 17:53:34.908188 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:34.908161    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp"]
Jul 28 17:53:36.537895 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:53:36.537856    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" path="/var/lib/kubelet/pods/3aa9cd9a-4a1d-4150-bc93-303ff34c6c63/volumes"
Jul 28 17:55:05.014524 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:05.014442    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:09.983444 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:09.983410    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:10.045561 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.045527    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"]
Jul 28 17:55:10.045853 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.045841    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kube-rbac-proxy"
Jul 28 17:55:10.045898 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.045854    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kube-rbac-proxy"
Jul 28 17:55:10.045898 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.045862    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="storage-initializer"
Jul 28 17:55:10.045898 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.045869    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="storage-initializer"
Jul 28 17:55:10.045898 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.045879    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container"
Jul 28 17:55:10.045898 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.045884    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container"
Jul 28 17:55:10.046084 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.045943    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kserve-container"
Jul 28 17:55:10.046084 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.045953    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa9cd9a-4a1d-4150-bc93-303ff34c6c63" containerName="kube-rbac-proxy"
Jul 28 17:55:10.048975 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.048956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.051737 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.051713    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 28 17:55:10.051847 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.051739    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:55:10.051847 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.051716    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:55:10.051847 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.051718    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:55:10.051847 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.051783    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:55:10.051847 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.051798    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:55:10.052721 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.052702    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:55:10.058165 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.058146    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"]
Jul 28 17:55:10.113722 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.113689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.113885 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.113745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.113885 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.113788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm9z5\" (UniqueName: \"kubernetes.io/projected/ec0e1426-1e37-43af-a013-ccc10261b333-kube-api-access-gm9z5\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.113885 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.113824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec0e1426-1e37-43af-a013-ccc10261b333-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.113992 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.113907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec0e1426-1e37-43af-a013-ccc10261b333-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.214453 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.214414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.214453 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.214457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9z5\" (UniqueName: \"kubernetes.io/projected/ec0e1426-1e37-43af-a013-ccc10261b333-kube-api-access-gm9z5\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.214700 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.214484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec0e1426-1e37-43af-a013-ccc10261b333-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.214700 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.214514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec0e1426-1e37-43af-a013-ccc10261b333-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.214700 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.214547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.215113 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.215088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec0e1426-1e37-43af-a013-ccc10261b333-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.215313 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.215295    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.215419 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.215397    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.217013 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.216995    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec0e1426-1e37-43af-a013-ccc10261b333-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.222437 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.222416    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9z5\" (UniqueName: \"kubernetes.io/projected/ec0e1426-1e37-43af-a013-ccc10261b333-kube-api-access-gm9z5\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.260165 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.260090    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:10.359248 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.359201    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:55:10.494862 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:10.494839    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"]
Jul 28 17:55:10.496757 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:55:10.496727    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0e1426_1e37_43af_a013_ccc10261b333.slice/crio-9441d5a9407a0a43f6988d0e38bd4ba0d77ab2d6006730d23a589cac4f47438a WatchSource:0}: Error finding container 9441d5a9407a0a43f6988d0e38bd4ba0d77ab2d6006730d23a589cac4f47438a: Status 404 returned error can't find the container with id 9441d5a9407a0a43f6988d0e38bd4ba0d77ab2d6006730d23a589cac4f47438a
Jul 28 17:55:11.297433 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:11.297400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm" event={"ID":"ec0e1426-1e37-43af-a013-ccc10261b333","Type":"ContainerStarted","Data":"6a67a14dfe269102659fdbe0cceb2af7703f4a95a85e17679c2eee37ec5d5a1c"}
Jul 28 17:55:11.297809 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:11.297456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm" event={"ID":"ec0e1426-1e37-43af-a013-ccc10261b333","Type":"ContainerStarted","Data":"9441d5a9407a0a43f6988d0e38bd4ba0d77ab2d6006730d23a589cac4f47438a"}
Jul 28 17:55:14.311632 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:14.311548    2583 generic.go:356] "Generic (PLEG): container finished" podID="ec0e1426-1e37-43af-a013-ccc10261b333" containerID="6a67a14dfe269102659fdbe0cceb2af7703f4a95a85e17679c2eee37ec5d5a1c" exitCode=0
Jul 28 17:55:14.311632 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:55:14.311619    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm" event={"ID":"ec0e1426-1e37-43af-a013-ccc10261b333","Type":"ContainerDied","Data":"6a67a14dfe269102659fdbe0cceb2af7703f4a95a85e17679c2eee37ec5d5a1c"}
Jul 28 17:57:23.013004 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:57:23.012970    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:57:23.017965 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:57:23.013265    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 17:57:23.988334 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:57:23.988291    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm" event={"ID":"ec0e1426-1e37-43af-a013-ccc10261b333","Type":"ContainerStarted","Data":"31099cc79d8d49a2572e2bcbc8f0e7955efea81f922a756717bf56fd23fe0b82"}
Jul 28 17:57:23.988534 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:57:23.988353    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm" event={"ID":"ec0e1426-1e37-43af-a013-ccc10261b333","Type":"ContainerStarted","Data":"41cb67bdd65d1aeff30b512c94ab0ad97f5b67422ce039055724d6f6a5b73621"}
Jul 28 17:57:23.988534 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:57:23.988388    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:57:23.988534 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:57:23.988409    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:57:24.024937 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:57:24.024726    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm" podStartSLOduration=1.359297152 podStartE2EDuration="2m14.024656168s" totalImagesPullingTime="2m9.217815001s" totalInitContainerRuntime="3.447544015s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:24.024179674 +0000 UTC m=+2410.037559568" watchObservedRunningTime="2026-07-28 17:57:24.024656168 +0000 UTC m=+2410.038036062"
Jul 28 17:57:29.999439 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:57:29.999406    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:58:00.004349 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:00.004267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:58:10.245762 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:10.245728    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"]
Jul 28 17:58:10.246170 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:10.245956    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm" podUID="ec0e1426-1e37-43af-a013-ccc10261b333" containerName="kserve-container" containerID="cri-o://41cb67bdd65d1aeff30b512c94ab0ad97f5b67422ce039055724d6f6a5b73621" gracePeriod=30
Jul 28 17:58:10.246170 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:10.246000    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm" podUID="ec0e1426-1e37-43af-a013-ccc10261b333" containerName="kube-rbac-proxy" containerID="cri-o://31099cc79d8d49a2572e2bcbc8f0e7955efea81f922a756717bf56fd23fe0b82" gracePeriod=30
Jul 28 17:58:10.299712 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:10.299630    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:11.205466 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.205430    2583 generic.go:356] "Generic (PLEG): container finished" podID="ec0e1426-1e37-43af-a013-ccc10261b333" containerID="31099cc79d8d49a2572e2bcbc8f0e7955efea81f922a756717bf56fd23fe0b82" exitCode=2
Jul 28 17:58:11.205466 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.205455    2583 generic.go:356] "Generic (PLEG): container finished" podID="ec0e1426-1e37-43af-a013-ccc10261b333" containerID="41cb67bdd65d1aeff30b512c94ab0ad97f5b67422ce039055724d6f6a5b73621" exitCode=0
Jul 28 17:58:11.205731 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.205501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm" event={"ID":"ec0e1426-1e37-43af-a013-ccc10261b333","Type":"ContainerDied","Data":"31099cc79d8d49a2572e2bcbc8f0e7955efea81f922a756717bf56fd23fe0b82"}
Jul 28 17:58:11.205731 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.205557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm" event={"ID":"ec0e1426-1e37-43af-a013-ccc10261b333","Type":"ContainerDied","Data":"41cb67bdd65d1aeff30b512c94ab0ad97f5b67422ce039055724d6f6a5b73621"}
Jul 28 17:58:11.293097 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.293073    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:58:11.343691 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.343610    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec0e1426-1e37-43af-a013-ccc10261b333-kserve-provision-location\") pod \"ec0e1426-1e37-43af-a013-ccc10261b333\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") "
Jul 28 17:58:11.343691 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.343644    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-cabundle-cert\") pod \"ec0e1426-1e37-43af-a013-ccc10261b333\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") "
Jul 28 17:58:11.343691 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.343665    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-kube-rbac-proxy-sar-config\") pod \"ec0e1426-1e37-43af-a013-ccc10261b333\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") "
Jul 28 17:58:11.343977 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.343707    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec0e1426-1e37-43af-a013-ccc10261b333-proxy-tls\") pod \"ec0e1426-1e37-43af-a013-ccc10261b333\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") "
Jul 28 17:58:11.343977 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.343831    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm9z5\" (UniqueName: \"kubernetes.io/projected/ec0e1426-1e37-43af-a013-ccc10261b333-kube-api-access-gm9z5\") pod \"ec0e1426-1e37-43af-a013-ccc10261b333\" (UID: \"ec0e1426-1e37-43af-a013-ccc10261b333\") "
Jul 28 17:58:11.344106 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.344054    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec0e1426-1e37-43af-a013-ccc10261b333" (UID: "ec0e1426-1e37-43af-a013-ccc10261b333"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:58:11.344106 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.344056    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0e1426-1e37-43af-a013-ccc10261b333-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec0e1426-1e37-43af-a013-ccc10261b333" (UID: "ec0e1426-1e37-43af-a013-ccc10261b333"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:58:11.344106 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.344071    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec0e1426-1e37-43af-a013-ccc10261b333" (UID: "ec0e1426-1e37-43af-a013-ccc10261b333"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:58:11.344219 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.344151    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec0e1426-1e37-43af-a013-ccc10261b333-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:58:11.344219 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.344171    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:58:11.344219 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.344185    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec0e1426-1e37-43af-a013-ccc10261b333-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:58:11.345764 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.345748    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec0e1426-1e37-43af-a013-ccc10261b333-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec0e1426-1e37-43af-a013-ccc10261b333" (UID: "ec0e1426-1e37-43af-a013-ccc10261b333"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:58:11.345858 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.345843    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0e1426-1e37-43af-a013-ccc10261b333-kube-api-access-gm9z5" (OuterVolumeSpecName: "kube-api-access-gm9z5") pod "ec0e1426-1e37-43af-a013-ccc10261b333" (UID: "ec0e1426-1e37-43af-a013-ccc10261b333"). InnerVolumeSpecName "kube-api-access-gm9z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:58:11.445624 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.445591    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec0e1426-1e37-43af-a013-ccc10261b333-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:58:11.445624 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:11.445627    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm9z5\" (UniqueName: \"kubernetes.io/projected/ec0e1426-1e37-43af-a013-ccc10261b333-kube-api-access-gm9z5\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:58:12.212735 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:12.212703    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"
Jul 28 17:58:12.212735 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:12.212716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm" event={"ID":"ec0e1426-1e37-43af-a013-ccc10261b333","Type":"ContainerDied","Data":"9441d5a9407a0a43f6988d0e38bd4ba0d77ab2d6006730d23a589cac4f47438a"}
Jul 28 17:58:12.212987 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:12.212822    2583 scope.go:117] "RemoveContainer" containerID="31099cc79d8d49a2572e2bcbc8f0e7955efea81f922a756717bf56fd23fe0b82"
Jul 28 17:58:12.221060 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:12.221010    2583 scope.go:117] "RemoveContainer" containerID="41cb67bdd65d1aeff30b512c94ab0ad97f5b67422ce039055724d6f6a5b73621"
Jul 28 17:58:12.228522 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:12.228507    2583 scope.go:117] "RemoveContainer" containerID="6a67a14dfe269102659fdbe0cceb2af7703f4a95a85e17679c2eee37ec5d5a1c"
Jul 28 17:58:12.236928 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:12.236909    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"]
Jul 28 17:58:12.240694 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:12.240670    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm"]
Jul 28 17:58:12.536761 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:12.536718    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec0e1426-1e37-43af-a013-ccc10261b333" path="/var/lib/kubelet/pods/ec0e1426-1e37-43af-a013-ccc10261b333/volumes"
Jul 28 17:58:14.611993 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:14.611962    2583 scope.go:117] "RemoveContainer" containerID="3b507f865e1c8bd6a8a20d4abc79dfcc5bb0d1595e344baffa275455feda5b3e"
Jul 28 17:58:14.619693 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:14.619671    2583 scope.go:117] "RemoveContainer" containerID="a22f2fba14dca9bb0181de596a86c0126e09d7d7dd8e738a2c365ac62404eb02"
Jul 28 17:58:14.626750 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:14.626729    2583 scope.go:117] "RemoveContainer" containerID="656c044bd9c6c3b3cb02a520460479ae1c597b7cea2d8084c2a696dd8e52e996"
Jul 28 17:58:15.372005 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.371970    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:15.448270 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.448236    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"]
Jul 28 17:58:15.448625 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.448611    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec0e1426-1e37-43af-a013-ccc10261b333" containerName="kserve-container"
Jul 28 17:58:15.448672 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.448628    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec0e1426-1e37-43af-a013-ccc10261b333" containerName="kserve-container"
Jul 28 17:58:15.448672 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.448640    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec0e1426-1e37-43af-a013-ccc10261b333" containerName="storage-initializer"
Jul 28 17:58:15.448672 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.448646    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec0e1426-1e37-43af-a013-ccc10261b333" containerName="storage-initializer"
Jul 28 17:58:15.448672 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.448655    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec0e1426-1e37-43af-a013-ccc10261b333" containerName="kube-rbac-proxy"
Jul 28 17:58:15.448672 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.448661    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec0e1426-1e37-43af-a013-ccc10261b333" containerName="kube-rbac-proxy"
Jul 28 17:58:15.448815 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.448722    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec0e1426-1e37-43af-a013-ccc10261b333" containerName="kserve-container"
Jul 28 17:58:15.448815 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.448732    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec0e1426-1e37-43af-a013-ccc10261b333" containerName="kube-rbac-proxy"
Jul 28 17:58:15.452195 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.452178    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.455390 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.455362    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 17:58:15.455635 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.455616    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:58:15.455707 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.455641    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:58:15.456693 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.456676    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 28 17:58:15.456693 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.456685    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:58:15.456818 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.456714    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:58:15.456818 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.456713    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:58:15.466158 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.466127    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"]
Jul 28 17:58:15.579686 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.579652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba79581e-d1e1-4360-8b55-e380707207ba-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.579686 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.579700    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.579955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.579735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnw26\" (UniqueName: \"kubernetes.io/projected/ba79581e-d1e1-4360-8b55-e380707207ba-kube-api-access-qnw26\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.579955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.579758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.579955 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.579900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba79581e-d1e1-4360-8b55-e380707207ba-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.680763 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.680675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba79581e-d1e1-4360-8b55-e380707207ba-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.680763 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.680719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.680763 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.680747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnw26\" (UniqueName: \"kubernetes.io/projected/ba79581e-d1e1-4360-8b55-e380707207ba-kube-api-access-qnw26\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.681292 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.680770    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.681292 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.680856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba79581e-d1e1-4360-8b55-e380707207ba-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.681292 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:58:15.680990    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jul 28 17:58:15.681292 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:58:15.681098    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba79581e-d1e1-4360-8b55-e380707207ba-proxy-tls podName:ba79581e-d1e1-4360-8b55-e380707207ba nodeName:}" failed. No retries permitted until 2026-07-28 17:58:16.181076507 +0000 UTC m=+2462.194456396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ba79581e-d1e1-4360-8b55-e380707207ba-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" (UID: "ba79581e-d1e1-4360-8b55-e380707207ba") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jul 28 17:58:15.681292 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.681173    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba79581e-d1e1-4360-8b55-e380707207ba-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.681489 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.681472    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.681489 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.681481    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.694799 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.694768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnw26\" (UniqueName: \"kubernetes.io/projected/ba79581e-d1e1-4360-8b55-e380707207ba-kube-api-access-qnw26\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:15.882647 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:15.882614    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:16.185198 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:16.185167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba79581e-d1e1-4360-8b55-e380707207ba-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:16.187509 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:16.187480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba79581e-d1e1-4360-8b55-e380707207ba-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:16.362737 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:16.362702    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:16.491743 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:16.491721    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"]
Jul 28 17:58:16.494043 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:58:16.494004    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba79581e_d1e1_4360_8b55_e380707207ba.slice/crio-fe1294d4f2606666cecd340eb81c530bb762c9ffc895c098caf4097d861ceeb0 WatchSource:0}: Error finding container fe1294d4f2606666cecd340eb81c530bb762c9ffc895c098caf4097d861ceeb0: Status 404 returned error can't find the container with id fe1294d4f2606666cecd340eb81c530bb762c9ffc895c098caf4097d861ceeb0
Jul 28 17:58:17.242217 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:17.242183    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" event={"ID":"ba79581e-d1e1-4360-8b55-e380707207ba","Type":"ContainerStarted","Data":"e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0"}
Jul 28 17:58:17.242677 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:17.242241    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" event={"ID":"ba79581e-d1e1-4360-8b55-e380707207ba","Type":"ContainerStarted","Data":"fe1294d4f2606666cecd340eb81c530bb762c9ffc895c098caf4097d861ceeb0"}
Jul 28 17:58:20.256542 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:20.256516    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba79581e-d1e1-4360-8b55-e380707207ba" containerID="e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0" exitCode=0
Jul 28 17:58:20.256835 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:20.256585    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" event={"ID":"ba79581e-d1e1-4360-8b55-e380707207ba","Type":"ContainerDied","Data":"e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0"}
Jul 28 17:58:21.263463 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:21.263429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" event={"ID":"ba79581e-d1e1-4360-8b55-e380707207ba","Type":"ContainerStarted","Data":"3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c"}
Jul 28 17:58:21.263885 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:21.263488    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" event={"ID":"ba79581e-d1e1-4360-8b55-e380707207ba","Type":"ContainerStarted","Data":"f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3"}
Jul 28 17:58:21.263885 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:21.263680    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:21.284981 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:21.284927    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" podStartSLOduration=2.583821086 podStartE2EDuration="6.284910019s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.701088933s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:21.283531802 +0000 UTC m=+2467.296911695" watchObservedRunningTime="2026-07-28 17:58:21.284910019 +0000 UTC m=+2467.298289912"
Jul 28 17:58:22.268675 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:22.268644    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:22.270033 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:22.269991    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 17:58:23.273458 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:23.273417    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 17:58:28.278315 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:28.278285    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:28.279618 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:28.279600    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:35.458753 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:35.458721    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"]
Jul 28 17:58:35.459247 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:35.458976    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="kserve-container" containerID="cri-o://f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3" gracePeriod=30
Jul 28 17:58:35.459247 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:35.459010    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="kube-rbac-proxy" containerID="cri-o://3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c" gracePeriod=30
Jul 28 17:58:35.483007 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:35.482979    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:36.108693 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.108668    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:36.264481 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.264449    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba79581e-d1e1-4360-8b55-e380707207ba-kserve-provision-location\") pod \"ba79581e-d1e1-4360-8b55-e380707207ba\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") "
Jul 28 17:58:36.264672 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.264501    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnw26\" (UniqueName: \"kubernetes.io/projected/ba79581e-d1e1-4360-8b55-e380707207ba-kube-api-access-qnw26\") pod \"ba79581e-d1e1-4360-8b55-e380707207ba\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") "
Jul 28 17:58:36.264672 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.264538    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-kube-rbac-proxy-sar-config\") pod \"ba79581e-d1e1-4360-8b55-e380707207ba\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") "
Jul 28 17:58:36.264672 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.264587    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-cabundle-cert\") pod \"ba79581e-d1e1-4360-8b55-e380707207ba\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") "
Jul 28 17:58:36.264672 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.264667    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba79581e-d1e1-4360-8b55-e380707207ba-proxy-tls\") pod \"ba79581e-d1e1-4360-8b55-e380707207ba\" (UID: \"ba79581e-d1e1-4360-8b55-e380707207ba\") "
Jul 28 17:58:36.264924 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.264876    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba79581e-d1e1-4360-8b55-e380707207ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba79581e-d1e1-4360-8b55-e380707207ba" (UID: "ba79581e-d1e1-4360-8b55-e380707207ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:58:36.264969 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.264938    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ba79581e-d1e1-4360-8b55-e380707207ba" (UID: "ba79581e-d1e1-4360-8b55-e380707207ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:58:36.264969 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.264949    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ba79581e-d1e1-4360-8b55-e380707207ba" (UID: "ba79581e-d1e1-4360-8b55-e380707207ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:58:36.266694 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.266665    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba79581e-d1e1-4360-8b55-e380707207ba-kube-api-access-qnw26" (OuterVolumeSpecName: "kube-api-access-qnw26") pod "ba79581e-d1e1-4360-8b55-e380707207ba" (UID: "ba79581e-d1e1-4360-8b55-e380707207ba"). InnerVolumeSpecName "kube-api-access-qnw26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:58:36.266822 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.266706    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba79581e-d1e1-4360-8b55-e380707207ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba79581e-d1e1-4360-8b55-e380707207ba" (UID: "ba79581e-d1e1-4360-8b55-e380707207ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:58:36.331604 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.331572    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba79581e-d1e1-4360-8b55-e380707207ba" containerID="3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c" exitCode=2
Jul 28 17:58:36.331604 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.331602    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba79581e-d1e1-4360-8b55-e380707207ba" containerID="f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3" exitCode=0
Jul 28 17:58:36.331795 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.331625    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"
Jul 28 17:58:36.331795 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.331655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" event={"ID":"ba79581e-d1e1-4360-8b55-e380707207ba","Type":"ContainerDied","Data":"3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c"}
Jul 28 17:58:36.331795 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.331716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" event={"ID":"ba79581e-d1e1-4360-8b55-e380707207ba","Type":"ContainerDied","Data":"f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3"}
Jul 28 17:58:36.331795 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.331737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt" event={"ID":"ba79581e-d1e1-4360-8b55-e380707207ba","Type":"ContainerDied","Data":"fe1294d4f2606666cecd340eb81c530bb762c9ffc895c098caf4097d861ceeb0"}
Jul 28 17:58:36.331795 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.331782    2583 scope.go:117] "RemoveContainer" containerID="3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c"
Jul 28 17:58:36.340012 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.339995    2583 scope.go:117] "RemoveContainer" containerID="f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3"
Jul 28 17:58:36.347372 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.347356    2583 scope.go:117] "RemoveContainer" containerID="e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0"
Jul 28 17:58:36.355103 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.355077    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"]
Jul 28 17:58:36.356162 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.356146    2583 scope.go:117] "RemoveContainer" containerID="3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c"
Jul 28 17:58:36.356436 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:58:36.356418    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c\": container with ID starting with 3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c not found: ID does not exist" containerID="3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c"
Jul 28 17:58:36.356480 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.356444    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c"} err="failed to get container status \"3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c\": rpc error: code = NotFound desc = could not find container \"3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c\": container with ID starting with 3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c not found: ID does not exist"
Jul 28 17:58:36.356480 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.356462    2583 scope.go:117] "RemoveContainer" containerID="f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3"
Jul 28 17:58:36.356708 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:58:36.356691    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3\": container with ID starting with f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3 not found: ID does not exist" containerID="f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3"
Jul 28 17:58:36.356787 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.356715    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3"} err="failed to get container status \"f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3\": rpc error: code = NotFound desc = could not find container \"f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3\": container with ID starting with f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3 not found: ID does not exist"
Jul 28 17:58:36.356787 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.356730    2583 scope.go:117] "RemoveContainer" containerID="e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0"
Jul 28 17:58:36.356979 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:58:36.356962    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0\": container with ID starting with e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0 not found: ID does not exist" containerID="e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0"
Jul 28 17:58:36.357154 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.356984    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0"} err="failed to get container status \"e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0\": rpc error: code = NotFound desc = could not find container \"e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0\": container with ID starting with e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0 not found: ID does not exist"
Jul 28 17:58:36.357154 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.356998    2583 scope.go:117] "RemoveContainer" containerID="3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c"
Jul 28 17:58:36.357283 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.357261    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c"} err="failed to get container status \"3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c\": rpc error: code = NotFound desc = could not find container \"3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c\": container with ID starting with 3373b1a323414e24a4897489c5604d85f6c831b5c1834c2cccf64a236e5d381c not found: ID does not exist"
Jul 28 17:58:36.357283 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.357278    2583 scope.go:117] "RemoveContainer" containerID="f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3"
Jul 28 17:58:36.357510 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.357489    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3"} err="failed to get container status \"f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3\": rpc error: code = NotFound desc = could not find container \"f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3\": container with ID starting with f9e63dc2626b7cbed9c2b947d72f68fefbf0862d0d456511458f186b963329e3 not found: ID does not exist"
Jul 28 17:58:36.357510 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.357510    2583 scope.go:117] "RemoveContainer" containerID="e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0"
Jul 28 17:58:36.357689 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.357674    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0"} err="failed to get container status \"e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0\": rpc error: code = NotFound desc = could not find container \"e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0\": container with ID starting with e3c61f384728533673f7009de1715ab51f7dc9489c4b75dbfb570861db8338c0 not found: ID does not exist"
Jul 28 17:58:36.363978 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.363957    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt"]
Jul 28 17:58:36.365646 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.365628    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:58:36.365703 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.365649    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba79581e-d1e1-4360-8b55-e380707207ba-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:58:36.365703 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.365658    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba79581e-d1e1-4360-8b55-e380707207ba-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:58:36.365703 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.365668    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnw26\" (UniqueName: \"kubernetes.io/projected/ba79581e-d1e1-4360-8b55-e380707207ba-kube-api-access-qnw26\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:58:36.365703 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.365677    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba79581e-d1e1-4360-8b55-e380707207ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:58:36.536910 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:36.536833    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" path="/var/lib/kubelet/pods/ba79581e-d1e1-4360-8b55-e380707207ba/volumes"
Jul 28 17:58:40.654673 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.654640    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:40.708670 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.708642    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"]
Jul 28 17:58:40.708984 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.708969    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="storage-initializer"
Jul 28 17:58:40.709069 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.708986    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="storage-initializer"
Jul 28 17:58:40.709069 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.709000    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="kserve-container"
Jul 28 17:58:40.709069 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.709005    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="kserve-container"
Jul 28 17:58:40.709069 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.709013    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="kube-rbac-proxy"
Jul 28 17:58:40.709069 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.709036    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="kube-rbac-proxy"
Jul 28 17:58:40.709237 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.709090    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="kube-rbac-proxy"
Jul 28 17:58:40.709237 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.709099    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba79581e-d1e1-4360-8b55-e380707207ba" containerName="kserve-container"
Jul 28 17:58:40.713350 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.713322    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.717131 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.717102    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:58:40.717794 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.717778    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:58:40.718006 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.717987    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:58:40.719009 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.718993    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:58:40.719113 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.719048    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:58:40.719113 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.719096    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 28 17:58:40.719227 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.719136    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:58:40.726417 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.726396    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"]
Jul 28 17:58:40.801301 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.801263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b77l\" (UniqueName: \"kubernetes.io/projected/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-api-access-5b77l\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.801482 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.801309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.801482 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.801344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff28ba6-d027-4ada-8b92-81b1515b0607-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.801482 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.801389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.801482 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.801452    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff28ba6-d027-4ada-8b92-81b1515b0607-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.902735 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.902704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b77l\" (UniqueName: \"kubernetes.io/projected/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-api-access-5b77l\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.902735 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.902740    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.902996 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.902763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff28ba6-d027-4ada-8b92-81b1515b0607-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.902996 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.902783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.902996 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.902807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff28ba6-d027-4ada-8b92-81b1515b0607-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.902996 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:58:40.902896    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 28 17:58:40.902996 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:58:40.902968    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dff28ba6-d027-4ada-8b92-81b1515b0607-proxy-tls podName:dff28ba6-d027-4ada-8b92-81b1515b0607 nodeName:}" failed. No retries permitted until 2026-07-28 17:58:41.402948717 +0000 UTC m=+2487.416328595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dff28ba6-d027-4ada-8b92-81b1515b0607-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" (UID: "dff28ba6-d027-4ada-8b92-81b1515b0607") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 28 17:58:40.903296 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.903229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff28ba6-d027-4ada-8b92-81b1515b0607-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.903512 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.903492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.903582 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.903492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:40.916267 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:40.916210    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b77l\" (UniqueName: \"kubernetes.io/projected/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-api-access-5b77l\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:41.079981 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:41.079931    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:41.407922 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:41.407879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff28ba6-d027-4ada-8b92-81b1515b0607-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:41.410291 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:41.410271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff28ba6-d027-4ada-8b92-81b1515b0607-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6z977\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:41.625317 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:41.625276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:41.837214 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:41.837181    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"]
Jul 28 17:58:41.840544 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:58:41.840516    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff28ba6_d027_4ada_8b92_81b1515b0607.slice/crio-65816e208b7f8397eee61447be6835fd4b71894bbeb9946cd00de036e80ae189 WatchSource:0}: Error finding container 65816e208b7f8397eee61447be6835fd4b71894bbeb9946cd00de036e80ae189: Status 404 returned error can't find the container with id 65816e208b7f8397eee61447be6835fd4b71894bbeb9946cd00de036e80ae189
Jul 28 17:58:42.361180 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:42.361141    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" event={"ID":"dff28ba6-d027-4ada-8b92-81b1515b0607","Type":"ContainerStarted","Data":"7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627"}
Jul 28 17:58:42.361421 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:42.361217    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" event={"ID":"dff28ba6-d027-4ada-8b92-81b1515b0607","Type":"ContainerStarted","Data":"65816e208b7f8397eee61447be6835fd4b71894bbeb9946cd00de036e80ae189"}
Jul 28 17:58:46.383824 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:46.383786    2583 generic.go:356] "Generic (PLEG): container finished" podID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerID="7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627" exitCode=0
Jul 28 17:58:46.383824 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:46.383827    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" event={"ID":"dff28ba6-d027-4ada-8b92-81b1515b0607","Type":"ContainerDied","Data":"7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627"}
Jul 28 17:58:47.392670 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:47.392635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" event={"ID":"dff28ba6-d027-4ada-8b92-81b1515b0607","Type":"ContainerStarted","Data":"0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215"}
Jul 28 17:58:47.393090 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:47.392697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" event={"ID":"dff28ba6-d027-4ada-8b92-81b1515b0607","Type":"ContainerStarted","Data":"dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135"}
Jul 28 17:58:47.393090 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:47.392912    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:47.393090 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:47.393080    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:58:47.430588 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:47.430537    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" podStartSLOduration=3.537910751 podStartE2EDuration="7.430520281s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.89260953s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:47.430218368 +0000 UTC m=+2493.443598266" watchObservedRunningTime="2026-07-28 17:58:47.430520281 +0000 UTC m=+2493.443900174"
Jul 28 17:58:53.402207 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:58:53.402177    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:59:23.406178 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:23.406145    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:59:30.791291 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:30.789934    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"]
Jul 28 17:59:30.791291 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:30.790372    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" podUID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerName="kserve-container" containerID="cri-o://dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135" gracePeriod=30
Jul 28 17:59:30.791291 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:30.790774    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" podUID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerName="kube-rbac-proxy" containerID="cri-o://0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215" gracePeriod=30
Jul 28 17:59:30.810484 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:30.810446    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:31.581820 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:31.581787    2583 generic.go:356] "Generic (PLEG): container finished" podID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerID="0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215" exitCode=2
Jul 28 17:59:31.582049 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:31.581864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" event={"ID":"dff28ba6-d027-4ada-8b92-81b1515b0607","Type":"ContainerDied","Data":"0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215"}
Jul 28 17:59:32.039884 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.039859    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:59:32.148790 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.148700    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff28ba6-d027-4ada-8b92-81b1515b0607-proxy-tls\") pod \"dff28ba6-d027-4ada-8b92-81b1515b0607\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") "
Jul 28 17:59:32.148790 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.148748    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b77l\" (UniqueName: \"kubernetes.io/projected/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-api-access-5b77l\") pod \"dff28ba6-d027-4ada-8b92-81b1515b0607\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") "
Jul 28 17:59:32.148790 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.148786    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-rbac-proxy-sar-config\") pod \"dff28ba6-d027-4ada-8b92-81b1515b0607\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") "
Jul 28 17:59:32.149087 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.148819    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-cabundle-cert\") pod \"dff28ba6-d027-4ada-8b92-81b1515b0607\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") "
Jul 28 17:59:32.149087 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.148984    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff28ba6-d027-4ada-8b92-81b1515b0607-kserve-provision-location\") pod \"dff28ba6-d027-4ada-8b92-81b1515b0607\" (UID: \"dff28ba6-d027-4ada-8b92-81b1515b0607\") "
Jul 28 17:59:32.149311 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.149287    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dff28ba6-d027-4ada-8b92-81b1515b0607" (UID: "dff28ba6-d027-4ada-8b92-81b1515b0607"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:59:32.149370 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.149291    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dff28ba6-d027-4ada-8b92-81b1515b0607" (UID: "dff28ba6-d027-4ada-8b92-81b1515b0607"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:59:32.149370 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.149335    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff28ba6-d027-4ada-8b92-81b1515b0607-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dff28ba6-d027-4ada-8b92-81b1515b0607" (UID: "dff28ba6-d027-4ada-8b92-81b1515b0607"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:32.150994 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.150969    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-api-access-5b77l" (OuterVolumeSpecName: "kube-api-access-5b77l") pod "dff28ba6-d027-4ada-8b92-81b1515b0607" (UID: "dff28ba6-d027-4ada-8b92-81b1515b0607"). InnerVolumeSpecName "kube-api-access-5b77l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:59:32.151168 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.151094    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff28ba6-d027-4ada-8b92-81b1515b0607-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dff28ba6-d027-4ada-8b92-81b1515b0607" (UID: "dff28ba6-d027-4ada-8b92-81b1515b0607"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:59:32.250200 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.250168    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff28ba6-d027-4ada-8b92-81b1515b0607-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:59:32.250200 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.250195    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b77l\" (UniqueName: \"kubernetes.io/projected/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-api-access-5b77l\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:59:32.250200 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.250204    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:59:32.250431 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.250213    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff28ba6-d027-4ada-8b92-81b1515b0607-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:59:32.250431 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.250222    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff28ba6-d027-4ada-8b92-81b1515b0607-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 17:59:32.588452 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.588420    2583 generic.go:356] "Generic (PLEG): container finished" podID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerID="dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135" exitCode=0
Jul 28 17:59:32.588615 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.588469    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" event={"ID":"dff28ba6-d027-4ada-8b92-81b1515b0607","Type":"ContainerDied","Data":"dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135"}
Jul 28 17:59:32.588615 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.588485    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"
Jul 28 17:59:32.588615 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.588524    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977" event={"ID":"dff28ba6-d027-4ada-8b92-81b1515b0607","Type":"ContainerDied","Data":"65816e208b7f8397eee61447be6835fd4b71894bbeb9946cd00de036e80ae189"}
Jul 28 17:59:32.588615 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.588551    2583 scope.go:117] "RemoveContainer" containerID="0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215"
Jul 28 17:59:32.596651 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.596635    2583 scope.go:117] "RemoveContainer" containerID="dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135"
Jul 28 17:59:32.604320 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.604300    2583 scope.go:117] "RemoveContainer" containerID="7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627"
Jul 28 17:59:32.613636 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.613607    2583 scope.go:117] "RemoveContainer" containerID="0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215"
Jul 28 17:59:32.613966 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:59:32.613937    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215\": container with ID starting with 0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215 not found: ID does not exist" containerID="0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215"
Jul 28 17:59:32.614029 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.613978    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215"} err="failed to get container status \"0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215\": rpc error: code = NotFound desc = could not find container \"0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215\": container with ID starting with 0883bbed02754e340e2eff24ae72c1227a3e7166d6e728bf49b3ca86cc2fe215 not found: ID does not exist"
Jul 28 17:59:32.614029 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.613998    2583 scope.go:117] "RemoveContainer" containerID="dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135"
Jul 28 17:59:32.614285 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:59:32.614268    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135\": container with ID starting with dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135 not found: ID does not exist" containerID="dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135"
Jul 28 17:59:32.614337 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.614291    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135"} err="failed to get container status \"dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135\": rpc error: code = NotFound desc = could not find container \"dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135\": container with ID starting with dc50aafb05d04f3a01fca8f471832475fb6e7f79855cd704eefad3b77a76b135 not found: ID does not exist"
Jul 28 17:59:32.614337 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.614307    2583 scope.go:117] "RemoveContainer" containerID="7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627"
Jul 28 17:59:32.614534 ip-10-0-132-11 kubenswrapper[2583]: E0728 17:59:32.614514    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627\": container with ID starting with 7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627 not found: ID does not exist" containerID="7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627"
Jul 28 17:59:32.614592 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.614543    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627"} err="failed to get container status \"7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627\": rpc error: code = NotFound desc = could not find container \"7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627\": container with ID starting with 7d21be3b1081febb445a91cf648787a3b15c5ce7411f3902d6a8cb6558d25627 not found: ID does not exist"
Jul 28 17:59:32.625203 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.625175    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"]
Jul 28 17:59:32.631749 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:32.631724    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977"]
Jul 28 17:59:34.538194 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:34.538159    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff28ba6-d027-4ada-8b92-81b1515b0607" path="/var/lib/kubelet/pods/dff28ba6-d027-4ada-8b92-81b1515b0607/volumes"
Jul 28 17:59:35.908958 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.908923    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:35.937159 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.937123    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"]
Jul 28 17:59:35.937455 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.937442    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerName="kube-rbac-proxy"
Jul 28 17:59:35.937455 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.937456    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerName="kube-rbac-proxy"
Jul 28 17:59:35.937540 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.937472    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerName="storage-initializer"
Jul 28 17:59:35.937540 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.937477    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerName="storage-initializer"
Jul 28 17:59:35.937540 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.937485    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerName="kserve-container"
Jul 28 17:59:35.937540 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.937490    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerName="kserve-container"
Jul 28 17:59:35.937674 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.937541    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerName="kserve-container"
Jul 28 17:59:35.937674 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.937552    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff28ba6-d027-4ada-8b92-81b1515b0607" containerName="kube-rbac-proxy"
Jul 28 17:59:35.945038 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.944999    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:35.947843 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.947815    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 28 17:59:35.947987 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.947868    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 17:59:35.947987 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.947828    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:59:35.948147 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.948131    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:59:35.948263 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.948247    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 28 17:59:35.948348 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.948262    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:59:35.948348 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.948249    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:59:35.952663 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.952634    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"]
Jul 28 17:59:35.982590 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.982556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:35.982742 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.982599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:35.982742 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.982625    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxzwg\" (UniqueName: \"kubernetes.io/projected/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-api-access-pxzwg\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:35.982742 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.982715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d844f691-9d0a-4a17-a79a-6a5fb309ff85-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:35.982879 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:35.982745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.074713 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.074681    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:36.083885 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.083850    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxzwg\" (UniqueName: \"kubernetes.io/projected/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-api-access-pxzwg\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.083989 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.083902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d844f691-9d0a-4a17-a79a-6a5fb309ff85-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.083989 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.083920    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.083989 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.083960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.084148 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.083993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.084472 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.084451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.084738 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.084713    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.084738 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.084725    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.086376 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.086360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d844f691-9d0a-4a17-a79a-6a5fb309ff85-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.092610 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.092589    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxzwg\" (UniqueName: \"kubernetes.io/projected/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-api-access-pxzwg\") pod \"isvc-sklearn-mcp-predictor-5996957568-pgdxp\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.257147 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.257105    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:36.389195 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.389169    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"]
Jul 28 17:59:36.391675 ip-10-0-132-11 kubenswrapper[2583]: W0728 17:59:36.391645    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd844f691_9d0a_4a17_a79a_6a5fb309ff85.slice/crio-7df4d513e4c7242ccc550aad71911cbde64115293439830ccfb7a8fa5fc683b8 WatchSource:0}: Error finding container 7df4d513e4c7242ccc550aad71911cbde64115293439830ccfb7a8fa5fc683b8: Status 404 returned error can't find the container with id 7df4d513e4c7242ccc550aad71911cbde64115293439830ccfb7a8fa5fc683b8
Jul 28 17:59:36.608803 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.608719    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" event={"ID":"d844f691-9d0a-4a17-a79a-6a5fb309ff85","Type":"ContainerStarted","Data":"cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152"}
Jul 28 17:59:36.608803 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:36.608787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" event={"ID":"d844f691-9d0a-4a17-a79a-6a5fb309ff85","Type":"ContainerStarted","Data":"7df4d513e4c7242ccc550aad71911cbde64115293439830ccfb7a8fa5fc683b8"}
Jul 28 17:59:40.627225 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:40.627192    2583 generic.go:356] "Generic (PLEG): container finished" podID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerID="cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152" exitCode=0
Jul 28 17:59:40.627643 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:40.627259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" event={"ID":"d844f691-9d0a-4a17-a79a-6a5fb309ff85","Type":"ContainerDied","Data":"cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152"}
Jul 28 17:59:40.688804 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:40.688781    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:59:41.636711 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:41.636619    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" event={"ID":"d844f691-9d0a-4a17-a79a-6a5fb309ff85","Type":"ContainerStarted","Data":"a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7"}
Jul 28 17:59:43.650255 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:43.650217    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" event={"ID":"d844f691-9d0a-4a17-a79a-6a5fb309ff85","Type":"ContainerStarted","Data":"b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc"}
Jul 28 17:59:43.650630 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:43.650292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" event={"ID":"d844f691-9d0a-4a17-a79a-6a5fb309ff85","Type":"ContainerStarted","Data":"375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2"}
Jul 28 17:59:43.650630 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:43.650335    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:43.650630 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:43.650370    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:43.650630 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:43.650403    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:43.679819 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:43.679761    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podStartSLOduration=2.33477476 podStartE2EDuration="8.679744939s" totalImagesPullingTime="2.788357609s" totalInitContainerRuntime="3.55661257s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:59:43.678463851 +0000 UTC m=+2549.691843744" watchObservedRunningTime="2026-07-28 17:59:43.679744939 +0000 UTC m=+2549.693124832"
Jul 28 17:59:49.660961 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:49.660930    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 17:59:58.864305 ip-10-0-132-11 kubenswrapper[2583]: I0728 17:59:58.864242    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:16.258294 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:16.258251    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 18:00:19.664997 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:19.664964    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 18:00:26.128031 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:26.127981    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:49.666564 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:49.666484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 18:00:56.048582 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:56.048547    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"]
Jul 28 18:00:56.048976 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:56.048873    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-container" containerID="cri-o://a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7" gracePeriod=30
Jul 28 18:00:56.048976 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:56.048907    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kube-rbac-proxy" containerID="cri-o://b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc" gracePeriod=30
Jul 28 18:00:56.049128 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:56.048964    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-agent" containerID="cri-o://375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2" gracePeriod=30
Jul 28 18:00:56.078736 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:56.078705    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:56.971922 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:56.971880    2583 generic.go:356] "Generic (PLEG): container finished" podID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerID="b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc" exitCode=2
Jul 28 18:00:56.972113 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:56.971952    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" event={"ID":"d844f691-9d0a-4a17-a79a-6a5fb309ff85","Type":"ContainerDied","Data":"b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc"}
Jul 28 18:00:58.983580 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:58.983547    2583 generic.go:356] "Generic (PLEG): container finished" podID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerID="a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7" exitCode=0
Jul 28 18:00:58.983951 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:58.983627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" event={"ID":"d844f691-9d0a-4a17-a79a-6a5fb309ff85","Type":"ContainerDied","Data":"a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7"}
Jul 28 18:00:59.655455 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:59.655409    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 18:00:59.661872 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:00:59.661845    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 18:01:01.599234 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:01.599205    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:01.703128 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:01.703094    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:04.655867 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:04.655824    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 18:01:09.656344 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:09.656295    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 18:01:09.656719 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:09.656444    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 18:01:09.661795 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:09.661766    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 18:01:14.656361 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:14.656321    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 18:01:19.655573 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:19.655532    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 18:01:19.662081 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:19.662050    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 18:01:19.662247 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:19.662232    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 18:01:24.655525 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:24.655485    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 18:01:26.233879 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.233855    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 18:01:26.356992 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.356911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d844f691-9d0a-4a17-a79a-6a5fb309ff85-proxy-tls\") pod \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") "
Jul 28 18:01:26.356992 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.356955    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-cabundle-cert\") pod \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") "
Jul 28 18:01:26.357224 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.357003    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxzwg\" (UniqueName: \"kubernetes.io/projected/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-api-access-pxzwg\") pod \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") "
Jul 28 18:01:26.357224 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.357076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-rbac-proxy-sar-config\") pod \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") "
Jul 28 18:01:26.357224 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.357110    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kserve-provision-location\") pod \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\" (UID: \"d844f691-9d0a-4a17-a79a-6a5fb309ff85\") "
Jul 28 18:01:26.357488 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.357386    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d844f691-9d0a-4a17-a79a-6a5fb309ff85" (UID: "d844f691-9d0a-4a17-a79a-6a5fb309ff85"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:26.357575 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.357460    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d844f691-9d0a-4a17-a79a-6a5fb309ff85" (UID: "d844f691-9d0a-4a17-a79a-6a5fb309ff85"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:26.357575 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.357490    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d844f691-9d0a-4a17-a79a-6a5fb309ff85" (UID: "d844f691-9d0a-4a17-a79a-6a5fb309ff85"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:01:26.359094 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.359073    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-api-access-pxzwg" (OuterVolumeSpecName: "kube-api-access-pxzwg") pod "d844f691-9d0a-4a17-a79a-6a5fb309ff85" (UID: "d844f691-9d0a-4a17-a79a-6a5fb309ff85"). InnerVolumeSpecName "kube-api-access-pxzwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:26.359155 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.359142    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d844f691-9d0a-4a17-a79a-6a5fb309ff85-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d844f691-9d0a-4a17-a79a-6a5fb309ff85" (UID: "d844f691-9d0a-4a17-a79a-6a5fb309ff85"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:01:26.458394 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.458363    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxzwg\" (UniqueName: \"kubernetes.io/projected/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-api-access-pxzwg\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:01:26.458394 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.458387    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:01:26.458394 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.458396    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d844f691-9d0a-4a17-a79a-6a5fb309ff85-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:01:26.458621 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.458406    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d844f691-9d0a-4a17-a79a-6a5fb309ff85-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:01:26.458621 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:26.458416    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d844f691-9d0a-4a17-a79a-6a5fb309ff85-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:01:27.103653 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.103593    2583 generic.go:356] "Generic (PLEG): container finished" podID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerID="375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2" exitCode=137
Jul 28 18:01:27.103878 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.103669    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"
Jul 28 18:01:27.103878 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.103677    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" event={"ID":"d844f691-9d0a-4a17-a79a-6a5fb309ff85","Type":"ContainerDied","Data":"375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2"}
Jul 28 18:01:27.103878 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.103800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp" event={"ID":"d844f691-9d0a-4a17-a79a-6a5fb309ff85","Type":"ContainerDied","Data":"7df4d513e4c7242ccc550aad71911cbde64115293439830ccfb7a8fa5fc683b8"}
Jul 28 18:01:27.103878 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.103848    2583 scope.go:117] "RemoveContainer" containerID="b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc"
Jul 28 18:01:27.112248 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.112231    2583 scope.go:117] "RemoveContainer" containerID="375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2"
Jul 28 18:01:27.119719 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.119698    2583 scope.go:117] "RemoveContainer" containerID="a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7"
Jul 28 18:01:27.127494 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.127469    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"]
Jul 28 18:01:27.127742 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.127726    2583 scope.go:117] "RemoveContainer" containerID="cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152"
Jul 28 18:01:27.132043 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.132007    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp"]
Jul 28 18:01:27.135252 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.135235    2583 scope.go:117] "RemoveContainer" containerID="b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc"
Jul 28 18:01:27.135498 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:01:27.135479    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc\": container with ID starting with b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc not found: ID does not exist" containerID="b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc"
Jul 28 18:01:27.135552 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.135509    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc"} err="failed to get container status \"b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc\": rpc error: code = NotFound desc = could not find container \"b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc\": container with ID starting with b56e7beade9fcc72727177ef9006f9ed1c3313b97db22f7c3a485d5d8ebb4abc not found: ID does not exist"
Jul 28 18:01:27.135552 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.135527    2583 scope.go:117] "RemoveContainer" containerID="375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2"
Jul 28 18:01:27.135769 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:01:27.135754    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2\": container with ID starting with 375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2 not found: ID does not exist" containerID="375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2"
Jul 28 18:01:27.135806 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.135775    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2"} err="failed to get container status \"375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2\": rpc error: code = NotFound desc = could not find container \"375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2\": container with ID starting with 375b9d1a099e5dec4ec6ba3dc034ef4b4d1bab092784e90714456dbe4a9677a2 not found: ID does not exist"
Jul 28 18:01:27.135806 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.135790    2583 scope.go:117] "RemoveContainer" containerID="a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7"
Jul 28 18:01:27.136049 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:01:27.136014    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7\": container with ID starting with a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7 not found: ID does not exist" containerID="a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7"
Jul 28 18:01:27.136100 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.136056    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7"} err="failed to get container status \"a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7\": rpc error: code = NotFound desc = could not find container \"a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7\": container with ID starting with a54ef5654f8a7cd6410f3a53a0d0a1f4b5fdaffd1bc0f24b4afcd642493020b7 not found: ID does not exist"
Jul 28 18:01:27.136100 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.136071    2583 scope.go:117] "RemoveContainer" containerID="cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152"
Jul 28 18:01:27.136273 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:01:27.136256    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152\": container with ID starting with cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152 not found: ID does not exist" containerID="cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152"
Jul 28 18:01:27.136329 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:27.136281    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152"} err="failed to get container status \"cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152\": rpc error: code = NotFound desc = could not find container \"cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152\": container with ID starting with cbf00e55d5db5fe8cabffc5c219648ef6ecb1d8efe431fcefc4cac8c48ffa152 not found: ID does not exist"
Jul 28 18:01:28.536498 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:01:28.536464    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" path="/var/lib/kubelet/pods/d844f691-9d0a-4a17-a79a-6a5fb309ff85/volumes"
Jul 28 18:02:13.015910 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:02:13.015877    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:18.018437 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:02:18.018409    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:19.206409 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:02:19.206376    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:23.037744 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:02:23.037641    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:02:23.040616 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:02:23.040379    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:03:19.694140 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:03:19.694099    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:24.853446 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:03:24.853409    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:25.007522 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:03:25.007482    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:26.826566 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:26.826529    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:32.018979 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.018947    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:32.055345 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055307    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"]
Jul 28 18:04:32.055705 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055689    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-agent"
Jul 28 18:04:32.055791 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055708    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-agent"
Jul 28 18:04:32.055791 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055724    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kube-rbac-proxy"
Jul 28 18:04:32.055791 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055732    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kube-rbac-proxy"
Jul 28 18:04:32.055791 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055755    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="storage-initializer"
Jul 28 18:04:32.055791 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055764    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="storage-initializer"
Jul 28 18:04:32.055791 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055783    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-container"
Jul 28 18:04:32.055791 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055790    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-container"
Jul 28 18:04:32.056185 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055863    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-agent"
Jul 28 18:04:32.056185 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055875    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kube-rbac-proxy"
Jul 28 18:04:32.056185 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.055886    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d844f691-9d0a-4a17-a79a-6a5fb309ff85" containerName="kserve-container"
Jul 28 18:04:32.058368 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.058347    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.061169 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.061147    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:04:32.061277 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.061171    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:04:32.061398 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.061299    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:04:32.061463 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.061402    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 28 18:04:32.062131 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.062114    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:04:32.062189 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.062117    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 28 18:04:32.062238 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.062199    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:04:32.073381 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.073355    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"]
Jul 28 18:04:32.217780 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.217749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.217974 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.217793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.217974 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.217813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hks5\" (UniqueName: \"kubernetes.io/projected/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-api-access-2hks5\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.217974 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.217831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.217974 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.217855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.236892 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.236862    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:32.319297 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.319193    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.319297 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.319248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.319539 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.319357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hks5\" (UniqueName: \"kubernetes.io/projected/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-api-access-2hks5\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.319539 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.319400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.319539 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.319440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.319801 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.319777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.319996 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.319977    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.320084 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.320007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.321828 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.321810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.328536 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.328511    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hks5\" (UniqueName: \"kubernetes.io/projected/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-api-access-2hks5\") pod \"isvc-pmml-predictor-8674795f59-tpxjh\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.369067 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.369011    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:32.505570 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.505478    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"]
Jul 28 18:04:32.507774 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:04:32.507742    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6d927c_15a6_40e4_86c8_3bcdb6c03f45.slice/crio-2b47db3a511a8499a8f812386fc6da3a3b13f08e4249f28d14b143d553f874db WatchSource:0}: Error finding container 2b47db3a511a8499a8f812386fc6da3a3b13f08e4249f28d14b143d553f874db: Status 404 returned error can't find the container with id 2b47db3a511a8499a8f812386fc6da3a3b13f08e4249f28d14b143d553f874db
Jul 28 18:04:32.916448 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.916410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" event={"ID":"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45","Type":"ContainerStarted","Data":"e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f"}
Jul 28 18:04:32.916654 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:32.916483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" event={"ID":"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45","Type":"ContainerStarted","Data":"2b47db3a511a8499a8f812386fc6da3a3b13f08e4249f28d14b143d553f874db"}
Jul 28 18:04:36.934170 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:36.934079    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerID="e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f" exitCode=0
Jul 28 18:04:36.934170 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:36.934133    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" event={"ID":"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45","Type":"ContainerDied","Data":"e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f"}
Jul 28 18:04:43.975229 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:43.975198    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" event={"ID":"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45","Type":"ContainerStarted","Data":"8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4"}
Jul 28 18:04:43.975591 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:43.975261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" event={"ID":"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45","Type":"ContainerStarted","Data":"0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d"}
Jul 28 18:04:43.975591 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:43.975444    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:43.999308 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:43.999260    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podStartSLOduration=1.182461204 podStartE2EDuration="11.99924674s" totalImagesPullingTime="6.784890776s" totalInitContainerRuntime="4.03189476s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:04:43.997390528 +0000 UTC m=+2850.010770422" watchObservedRunningTime="2026-07-28 18:04:43.99924674 +0000 UTC m=+2850.012626632"
Jul 28 18:04:44.979803 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:44.979771    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:44.981119 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:44.981093    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 18:04:45.983736 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:45.983687    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 18:04:50.988946 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:50.988918    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:04:50.989446 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:04:50.989418    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 18:05:00.989936 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:05:00.989899    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 18:05:10.990245 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:05:10.990201    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 18:05:20.990174 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:05:20.990134    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 18:05:30.989759 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:05:30.989718    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 18:05:40.990065 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:05:40.990005    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 18:05:50.990209 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:05:50.990168    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 18:05:57.532933 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:05:57.532889    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 18:06:07.533999 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:07.533964    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:06:13.389218 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:13.389184    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"]
Jul 28 18:06:13.389698 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:13.389439    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container" containerID="cri-o://0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d" gracePeriod=30
Jul 28 18:06:13.389698 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:13.389486    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kube-rbac-proxy" containerID="cri-o://8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4" gracePeriod=30
Jul 28 18:06:13.509796 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:13.509765    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:14.367701 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:14.367670    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerID="8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4" exitCode=2
Jul 28 18:06:14.367878 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:14.367717    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" event={"ID":"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45","Type":"ContainerDied","Data":"8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4"}
Jul 28 18:06:15.984262 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:15.984219    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 28 18:06:17.140576 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.140553    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:06:17.303309 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.303273    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-rbac-proxy-sar-config\") pod \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") "
Jul 28 18:06:17.303509 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.303330    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hks5\" (UniqueName: \"kubernetes.io/projected/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-api-access-2hks5\") pod \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") "
Jul 28 18:06:17.303509 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.303352    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-cabundle-cert\") pod \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") "
Jul 28 18:06:17.303509 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.303403    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kserve-provision-location\") pod \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") "
Jul 28 18:06:17.303509 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.303431    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-proxy-tls\") pod \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\" (UID: \"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45\") "
Jul 28 18:06:17.303745 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.303692    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" (UID: "4e6d927c-15a6-40e4-86c8-3bcdb6c03f45"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:06:17.303745 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.303722    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" (UID: "4e6d927c-15a6-40e4-86c8-3bcdb6c03f45"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:06:17.303844 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.303780    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" (UID: "4e6d927c-15a6-40e4-86c8-3bcdb6c03f45"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:06:17.305581 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.305551    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" (UID: "4e6d927c-15a6-40e4-86c8-3bcdb6c03f45"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:06:17.305725 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.305702    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-api-access-2hks5" (OuterVolumeSpecName: "kube-api-access-2hks5") pod "4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" (UID: "4e6d927c-15a6-40e4-86c8-3bcdb6c03f45"). InnerVolumeSpecName "kube-api-access-2hks5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:06:17.382645 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.382611    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerID="0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d" exitCode=0
Jul 28 18:06:17.382825 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.382670    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"
Jul 28 18:06:17.382825 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.382676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" event={"ID":"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45","Type":"ContainerDied","Data":"0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d"}
Jul 28 18:06:17.382825 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.382726    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh" event={"ID":"4e6d927c-15a6-40e4-86c8-3bcdb6c03f45","Type":"ContainerDied","Data":"2b47db3a511a8499a8f812386fc6da3a3b13f08e4249f28d14b143d553f874db"}
Jul 28 18:06:17.382825 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.382752    2583 scope.go:117] "RemoveContainer" containerID="8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4"
Jul 28 18:06:17.391137 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.391094    2583 scope.go:117] "RemoveContainer" containerID="0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d"
Jul 28 18:06:17.398798 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.398777    2583 scope.go:117] "RemoveContainer" containerID="e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f"
Jul 28 18:06:17.404070 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.404048    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:06:17.404162 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.404071    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:06:17.404162 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.404083    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:06:17.404162 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.404093    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hks5\" (UniqueName: \"kubernetes.io/projected/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-kube-api-access-2hks5\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:06:17.404162 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.404101    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:06:17.406005 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.405992    2583 scope.go:117] "RemoveContainer" containerID="8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4"
Jul 28 18:06:17.406304 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:06:17.406279    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4\": container with ID starting with 8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4 not found: ID does not exist" containerID="8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4"
Jul 28 18:06:17.406409 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.406318    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4"} err="failed to get container status \"8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4\": rpc error: code = NotFound desc = could not find container \"8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4\": container with ID starting with 8723ece61587acd259b00211e3ad9fc6f3320e34064d29c714de1635bf4b52c4 not found: ID does not exist"
Jul 28 18:06:17.406409 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.406345    2583 scope.go:117] "RemoveContainer" containerID="0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d"
Jul 28 18:06:17.406656 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:06:17.406631    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d\": container with ID starting with 0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d not found: ID does not exist" containerID="0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d"
Jul 28 18:06:17.406741 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.406667    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d"} err="failed to get container status \"0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d\": rpc error: code = NotFound desc = could not find container \"0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d\": container with ID starting with 0847a696bb39f72467ae44d9d8959e54c3e72741a4fce5c1cb621d8606eb635d not found: ID does not exist"
Jul 28 18:06:17.406741 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.406690    2583 scope.go:117] "RemoveContainer" containerID="e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f"
Jul 28 18:06:17.406985 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:06:17.406957    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f\": container with ID starting with e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f not found: ID does not exist" containerID="e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f"
Jul 28 18:06:17.407127 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.406992    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f"} err="failed to get container status \"e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f\": rpc error: code = NotFound desc = could not find container \"e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f\": container with ID starting with e2a6091765c6e075bb4e57bc5407b0216fdd9f3ca466b3ddfefc187ae61ede9f not found: ID does not exist"
Jul 28 18:06:17.409447 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.409424    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"]
Jul 28 18:06:17.421072 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:17.421049    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh"]
Jul 28 18:06:18.395291 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.395255    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:18.478831 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.478797    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"]
Jul 28 18:06:18.479142 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.479130    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kube-rbac-proxy"
Jul 28 18:06:18.479195 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.479144    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kube-rbac-proxy"
Jul 28 18:06:18.479195 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.479161    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="storage-initializer"
Jul 28 18:06:18.479195 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.479167    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="storage-initializer"
Jul 28 18:06:18.479195 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.479176    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container"
Jul 28 18:06:18.479195 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.479182    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container"
Jul 28 18:06:18.479350 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.479235    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kserve-container"
Jul 28 18:06:18.479350 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.479241    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" containerName="kube-rbac-proxy"
Jul 28 18:06:18.482291 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.482274    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.485086 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.485070    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:06:18.485352 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.485325    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:06:18.485352 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.485343    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:06:18.485522 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.485340    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:06:18.486150 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.486135    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:06:18.486254 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.486170    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:06:18.486254 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.486210    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 28 18:06:18.500691 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.500669    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"]
Jul 28 18:06:18.532443 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.532375    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:18.537701 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.537679    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6d927c-15a6-40e4-86c8-3bcdb6c03f45" path="/var/lib/kubelet/pods/4e6d927c-15a6-40e4-86c8-3bcdb6c03f45/volumes"
Jul 28 18:06:18.613177 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.613145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.613177 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.613190    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.613396 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.613210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75tnl\" (UniqueName: \"kubernetes.io/projected/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-api-access-75tnl\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.613396 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.613260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.613396 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.613317    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.714219 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.714130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.714219 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.714173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.714219 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.714212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.714492 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.714235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75tnl\" (UniqueName: \"kubernetes.io/projected/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-api-access-75tnl\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.714492 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.714259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.714708 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.714683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.714866 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.714848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.714938 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.714905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.716688 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.716667    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.723942 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.723920    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75tnl\" (UniqueName: \"kubernetes.io/projected/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-api-access-75tnl\") pod \"isvc-pmml-runtime-predictor-885fc8777-87tld\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.793194 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.793146    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:18.925982 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:18.925948    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"]
Jul 28 18:06:18.928923 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:06:18.928899    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ff3f7e_0921_4838_bcac_5a7b97f018b6.slice/crio-386226a90a0e76f252b9b8370f808dd5266fa907dcb7aecd6a0ae4a8482b4f41 WatchSource:0}: Error finding container 386226a90a0e76f252b9b8370f808dd5266fa907dcb7aecd6a0ae4a8482b4f41: Status 404 returned error can't find the container with id 386226a90a0e76f252b9b8370f808dd5266fa907dcb7aecd6a0ae4a8482b4f41
Jul 28 18:06:19.395704 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:19.395624    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" event={"ID":"c7ff3f7e-0921-4838-bcac-5a7b97f018b6","Type":"ContainerStarted","Data":"ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d"}
Jul 28 18:06:19.395704 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:19.395682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" event={"ID":"c7ff3f7e-0921-4838-bcac-5a7b97f018b6","Type":"ContainerStarted","Data":"386226a90a0e76f252b9b8370f808dd5266fa907dcb7aecd6a0ae4a8482b4f41"}
Jul 28 18:06:23.414500 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:23.414466    2583 generic.go:356] "Generic (PLEG): container finished" podID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerID="ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d" exitCode=0
Jul 28 18:06:23.414874 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:23.414531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" event={"ID":"c7ff3f7e-0921-4838-bcac-5a7b97f018b6","Type":"ContainerDied","Data":"ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d"}
Jul 28 18:06:24.421306 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:24.421270    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" event={"ID":"c7ff3f7e-0921-4838-bcac-5a7b97f018b6","Type":"ContainerStarted","Data":"0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c"}
Jul 28 18:06:24.421685 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:24.421334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" event={"ID":"c7ff3f7e-0921-4838-bcac-5a7b97f018b6","Type":"ContainerStarted","Data":"98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1"}
Jul 28 18:06:24.421685 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:24.421514    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:25.425235 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:25.425198    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:25.426544 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:25.426515    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 18:06:26.429312 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:26.429268    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 18:06:31.434104 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:31.434074    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:06:31.434722 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:31.434698    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 18:06:31.463721 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:31.463666    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podStartSLOduration=9.629963047 podStartE2EDuration="13.463650886s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.833687839s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:06:24.443804599 +0000 UTC m=+2950.457184497" watchObservedRunningTime="2026-07-28 18:06:31.463650886 +0000 UTC m=+2957.477030792"
Jul 28 18:06:37.316176 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:37.316090    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:41.435540 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:41.435490    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 18:06:51.434857 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:06:51.434812    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 18:07:01.435091 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:01.435041    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 18:07:11.435530 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:11.435488    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 18:07:21.435466 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:21.435413    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 18:07:23.064099 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:23.063969    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:07:23.066301 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:23.066277    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:07:31.434816 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:31.434772    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 18:07:41.435441 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:41.435402    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 18:07:44.538011 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:44.537984    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:07:49.426299 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:49.426265    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:49.612262 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:49.612228    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"]
Jul 28 18:07:49.612488 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:49.612452    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container" containerID="cri-o://98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1" gracePeriod=30
Jul 28 18:07:49.612534 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:49.612501    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kube-rbac-proxy" containerID="cri-o://0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c" gracePeriod=30
Jul 28 18:07:49.619282 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:49.619257    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:49.790148 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:49.790116    2583 generic.go:356] "Generic (PLEG): container finished" podID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerID="0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c" exitCode=2
Jul 28 18:07:49.790333 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:49.790197    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" event={"ID":"c7ff3f7e-0921-4838-bcac-5a7b97f018b6","Type":"ContainerDied","Data":"0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c"}
Jul 28 18:07:51.429740 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:51.429688    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 18:07:53.453360 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.453337    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:07:53.487886 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.487826    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kserve-provision-location\") pod \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") "
Jul 28 18:07:53.487886 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.487868    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75tnl\" (UniqueName: \"kubernetes.io/projected/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-api-access-75tnl\") pod \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") "
Jul 28 18:07:53.488179 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.487888    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-rbac-proxy-sar-config\") pod \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") "
Jul 28 18:07:53.488179 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.487927    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-cabundle-cert\") pod \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") "
Jul 28 18:07:53.488179 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.487962    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-proxy-tls\") pod \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\" (UID: \"c7ff3f7e-0921-4838-bcac-5a7b97f018b6\") "
Jul 28 18:07:53.488350 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.488191    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7ff3f7e-0921-4838-bcac-5a7b97f018b6" (UID: "c7ff3f7e-0921-4838-bcac-5a7b97f018b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:07:53.488402 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.488338    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c7ff3f7e-0921-4838-bcac-5a7b97f018b6" (UID: "c7ff3f7e-0921-4838-bcac-5a7b97f018b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:07:53.488402 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.488346    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7ff3f7e-0921-4838-bcac-5a7b97f018b6" (UID: "c7ff3f7e-0921-4838-bcac-5a7b97f018b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:07:53.489905 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.489885    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-api-access-75tnl" (OuterVolumeSpecName: "kube-api-access-75tnl") pod "c7ff3f7e-0921-4838-bcac-5a7b97f018b6" (UID: "c7ff3f7e-0921-4838-bcac-5a7b97f018b6"). InnerVolumeSpecName "kube-api-access-75tnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:07:53.490010 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.489993    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c7ff3f7e-0921-4838-bcac-5a7b97f018b6" (UID: "c7ff3f7e-0921-4838-bcac-5a7b97f018b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:07:53.589821 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.589442    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:07:53.589821 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.589626    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:07:53.589821 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.589782    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:07:53.589821 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.589800    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75tnl\" (UniqueName: \"kubernetes.io/projected/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-api-access-75tnl\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:07:53.589821 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.589816    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7ff3f7e-0921-4838-bcac-5a7b97f018b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:07:53.807780 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.807701    2583 generic.go:356] "Generic (PLEG): container finished" podID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerID="98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1" exitCode=0
Jul 28 18:07:53.807780 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.807756    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"
Jul 28 18:07:53.807780 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.807764    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" event={"ID":"c7ff3f7e-0921-4838-bcac-5a7b97f018b6","Type":"ContainerDied","Data":"98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1"}
Jul 28 18:07:53.808049 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.807813    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld" event={"ID":"c7ff3f7e-0921-4838-bcac-5a7b97f018b6","Type":"ContainerDied","Data":"386226a90a0e76f252b9b8370f808dd5266fa907dcb7aecd6a0ae4a8482b4f41"}
Jul 28 18:07:53.808049 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.807841    2583 scope.go:117] "RemoveContainer" containerID="0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c"
Jul 28 18:07:53.816390 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.816371    2583 scope.go:117] "RemoveContainer" containerID="98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1"
Jul 28 18:07:53.823655 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.823639    2583 scope.go:117] "RemoveContainer" containerID="ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d"
Jul 28 18:07:53.830727 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.830702    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"]
Jul 28 18:07:53.830965 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.830950    2583 scope.go:117] "RemoveContainer" containerID="0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c"
Jul 28 18:07:53.831348 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:07:53.831327    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c\": container with ID starting with 0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c not found: ID does not exist" containerID="0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c"
Jul 28 18:07:53.831439 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.831359    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c"} err="failed to get container status \"0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c\": rpc error: code = NotFound desc = could not find container \"0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c\": container with ID starting with 0fb309eb6b66e0838b509aae7db484811744f7578a1f6dac348cfab83a92774c not found: ID does not exist"
Jul 28 18:07:53.831439 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.831383    2583 scope.go:117] "RemoveContainer" containerID="98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1"
Jul 28 18:07:53.831682 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:07:53.831665    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1\": container with ID starting with 98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1 not found: ID does not exist" containerID="98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1"
Jul 28 18:07:53.831722 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.831688    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1"} err="failed to get container status \"98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1\": rpc error: code = NotFound desc = could not find container \"98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1\": container with ID starting with 98eeedc968bd68ae9d9211074f71182c309e0d1aa6e3714b0669963991469dd1 not found: ID does not exist"
Jul 28 18:07:53.831722 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.831709    2583 scope.go:117] "RemoveContainer" containerID="ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d"
Jul 28 18:07:53.831916 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:07:53.831896    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d\": container with ID starting with ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d not found: ID does not exist" containerID="ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d"
Jul 28 18:07:53.831969 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.831922    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d"} err="failed to get container status \"ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d\": rpc error: code = NotFound desc = could not find container \"ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d\": container with ID starting with ce9d45c806f6a446e7cddda234b4c39c073e4f8ce267782d5686409be455f16d not found: ID does not exist"
Jul 28 18:07:53.836034 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:53.836004    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld"]
Jul 28 18:07:54.536658 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.536622    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" path="/var/lib/kubelet/pods/c7ff3f7e-0921-4838-bcac-5a7b97f018b6/volumes"
Jul 28 18:07:54.782858 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.778527    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:54.833759 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.833671    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"]
Jul 28 18:07:54.834126 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.834100    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container"
Jul 28 18:07:54.834126 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.834123    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container"
Jul 28 18:07:54.834302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.834137    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kube-rbac-proxy"
Jul 28 18:07:54.834302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.834143    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kube-rbac-proxy"
Jul 28 18:07:54.834302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.834154    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="storage-initializer"
Jul 28 18:07:54.834302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.834159    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="storage-initializer"
Jul 28 18:07:54.834302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.834234    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kube-rbac-proxy"
Jul 28 18:07:54.834302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.834246    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7ff3f7e-0921-4838-bcac-5a7b97f018b6" containerName="kserve-container"
Jul 28 18:07:54.837474 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.837457    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:54.840204 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.840184    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:07:54.840317 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.840184    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 18:07:54.840317 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.840270    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:07:54.841092 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.841053    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:07:54.841223 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.841129    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:07:54.841223 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.841219    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 28 18:07:54.841330 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.841222    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:07:54.850836 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.850816    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"]
Jul 28 18:07:54.900400 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.900370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28mc\" (UniqueName: \"kubernetes.io/projected/ab5801bd-e79c-415e-9370-52cf6be18501-kube-api-access-k28mc\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:54.900566 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.900415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab5801bd-e79c-415e-9370-52cf6be18501-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:54.900566 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.900443    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:54.900566 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.900513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:54.900677 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:54.900602    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab5801bd-e79c-415e-9370-52cf6be18501-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.001599 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.001567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.001806 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.001649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab5801bd-e79c-415e-9370-52cf6be18501-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.001806 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.001683    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28mc\" (UniqueName: \"kubernetes.io/projected/ab5801bd-e79c-415e-9370-52cf6be18501-kube-api-access-k28mc\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.001806 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.001722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab5801bd-e79c-415e-9370-52cf6be18501-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.001990 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.001910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.002115 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.002096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab5801bd-e79c-415e-9370-52cf6be18501-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.002408 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.002378    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.002514 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.002471    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.004199 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.004182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab5801bd-e79c-415e-9370-52cf6be18501-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.019568 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.019537    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28mc\" (UniqueName: \"kubernetes.io/projected/ab5801bd-e79c-415e-9370-52cf6be18501-kube-api-access-k28mc\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.039254 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.039217    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:55.147637 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.147545    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:07:55.281040 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.281001    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"]
Jul 28 18:07:55.283514 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:07:55.283484    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5801bd_e79c_415e_9370_52cf6be18501.slice/crio-0db67fc2b31e28695cf27cf96610efb9e16110f09a96e12a7b7432b337b66eb7 WatchSource:0}: Error finding container 0db67fc2b31e28695cf27cf96610efb9e16110f09a96e12a7b7432b337b66eb7: Status 404 returned error can't find the container with id 0db67fc2b31e28695cf27cf96610efb9e16110f09a96e12a7b7432b337b66eb7
Jul 28 18:07:55.820417 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.820377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" event={"ID":"ab5801bd-e79c-415e-9370-52cf6be18501","Type":"ContainerStarted","Data":"e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb"}
Jul 28 18:07:55.820901 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:55.820456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" event={"ID":"ab5801bd-e79c-415e-9370-52cf6be18501","Type":"ContainerStarted","Data":"0db67fc2b31e28695cf27cf96610efb9e16110f09a96e12a7b7432b337b66eb7"}
Jul 28 18:07:59.838325 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:59.838294    2583 generic.go:356] "Generic (PLEG): container finished" podID="ab5801bd-e79c-415e-9370-52cf6be18501" containerID="e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb" exitCode=0
Jul 28 18:07:59.838789 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:07:59.838358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" event={"ID":"ab5801bd-e79c-415e-9370-52cf6be18501","Type":"ContainerDied","Data":"e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb"}
Jul 28 18:08:00.844953 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:00.844919    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" event={"ID":"ab5801bd-e79c-415e-9370-52cf6be18501","Type":"ContainerStarted","Data":"4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35"}
Jul 28 18:08:00.845471 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:00.845003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" event={"ID":"ab5801bd-e79c-415e-9370-52cf6be18501","Type":"ContainerStarted","Data":"93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd"}
Jul 28 18:08:00.845471 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:00.845303    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:08:00.845471 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:00.845448    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:08:00.846567 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:00.846543    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 18:08:00.864860 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:00.864813    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podStartSLOduration=3.204405171 podStartE2EDuration="6.864799947s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.660394776s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:08:00.863470476 +0000 UTC m=+3046.876850360" watchObservedRunningTime="2026-07-28 18:08:00.864799947 +0000 UTC m=+3046.878179839"
Jul 28 18:08:01.850108 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:01.850071    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 18:08:06.855964 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:06.855887    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:08:06.856425 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:06.856379    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 18:08:16.856602 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:16.856563    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 18:08:21.837636 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:21.837600    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:23.602405 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:23.602359    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:26.856805 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:26.856764    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 18:08:36.856516 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:36.856477    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 18:08:46.856788 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:46.856746    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 18:08:56.857323 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:08:56.857283    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 18:09:06.856432 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:06.856386    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 18:09:16.857190 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:16.857160    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:09:25.950898 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:25.950866    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"]
Jul 28 18:09:25.951424 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:25.951100    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container" containerID="cri-o://93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd" gracePeriod=30
Jul 28 18:09:25.951424 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:25.951173    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kube-rbac-proxy" containerID="cri-o://4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35" gracePeriod=30
Jul 28 18:09:25.967046 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:25.966999    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:09:26.221060 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:26.220969    2583 generic.go:356] "Generic (PLEG): container finished" podID="ab5801bd-e79c-415e-9370-52cf6be18501" containerID="4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35" exitCode=2
Jul 28 18:09:26.221060 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:26.221049    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" event={"ID":"ab5801bd-e79c-415e-9370-52cf6be18501","Type":"ContainerDied","Data":"4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35"}
Jul 28 18:09:26.851297 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:26.851256    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 18:09:26.856653 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:26.856625    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 18:09:29.699299 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.699273    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:09:29.791803 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.791705    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-cabundle-cert\") pod \"ab5801bd-e79c-415e-9370-52cf6be18501\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") "
Jul 28 18:09:29.791803 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.791749    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab5801bd-e79c-415e-9370-52cf6be18501-proxy-tls\") pod \"ab5801bd-e79c-415e-9370-52cf6be18501\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") "
Jul 28 18:09:29.791803 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.791771    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k28mc\" (UniqueName: \"kubernetes.io/projected/ab5801bd-e79c-415e-9370-52cf6be18501-kube-api-access-k28mc\") pod \"ab5801bd-e79c-415e-9370-52cf6be18501\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") "
Jul 28 18:09:29.792138 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.791809    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab5801bd-e79c-415e-9370-52cf6be18501-kserve-provision-location\") pod \"ab5801bd-e79c-415e-9370-52cf6be18501\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") "
Jul 28 18:09:29.792138 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.791854    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-kube-rbac-proxy-sar-config\") pod \"ab5801bd-e79c-415e-9370-52cf6be18501\" (UID: \"ab5801bd-e79c-415e-9370-52cf6be18501\") "
Jul 28 18:09:29.792249 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.792171    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5801bd-e79c-415e-9370-52cf6be18501-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab5801bd-e79c-415e-9370-52cf6be18501" (UID: "ab5801bd-e79c-415e-9370-52cf6be18501"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:09:29.792249 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.792199    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab5801bd-e79c-415e-9370-52cf6be18501" (UID: "ab5801bd-e79c-415e-9370-52cf6be18501"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:09:29.792325 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.792252    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab5801bd-e79c-415e-9370-52cf6be18501" (UID: "ab5801bd-e79c-415e-9370-52cf6be18501"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:09:29.793885 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.793851    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5801bd-e79c-415e-9370-52cf6be18501-kube-api-access-k28mc" (OuterVolumeSpecName: "kube-api-access-k28mc") pod "ab5801bd-e79c-415e-9370-52cf6be18501" (UID: "ab5801bd-e79c-415e-9370-52cf6be18501"). InnerVolumeSpecName "kube-api-access-k28mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:09:29.793989 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.793931    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5801bd-e79c-415e-9370-52cf6be18501-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab5801bd-e79c-415e-9370-52cf6be18501" (UID: "ab5801bd-e79c-415e-9370-52cf6be18501"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:09:29.892620 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.892585    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:09:29.892620 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.892614    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab5801bd-e79c-415e-9370-52cf6be18501-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:09:29.892620 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.892623    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab5801bd-e79c-415e-9370-52cf6be18501-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:09:29.892856 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.892631    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k28mc\" (UniqueName: \"kubernetes.io/projected/ab5801bd-e79c-415e-9370-52cf6be18501-kube-api-access-k28mc\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:09:29.892856 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:29.892641    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab5801bd-e79c-415e-9370-52cf6be18501-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:09:30.239206 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.239169    2583 generic.go:356] "Generic (PLEG): container finished" podID="ab5801bd-e79c-415e-9370-52cf6be18501" containerID="93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd" exitCode=0
Jul 28 18:09:30.239395 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.239225    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"
Jul 28 18:09:30.239395 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.239245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" event={"ID":"ab5801bd-e79c-415e-9370-52cf6be18501","Type":"ContainerDied","Data":"93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd"}
Jul 28 18:09:30.239395 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.239304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf" event={"ID":"ab5801bd-e79c-415e-9370-52cf6be18501","Type":"ContainerDied","Data":"0db67fc2b31e28695cf27cf96610efb9e16110f09a96e12a7b7432b337b66eb7"}
Jul 28 18:09:30.239395 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.239334    2583 scope.go:117] "RemoveContainer" containerID="4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35"
Jul 28 18:09:30.249650 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.249624    2583 scope.go:117] "RemoveContainer" containerID="93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd"
Jul 28 18:09:30.257954 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.257937    2583 scope.go:117] "RemoveContainer" containerID="e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb"
Jul 28 18:09:30.266082 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.266055    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"]
Jul 28 18:09:30.266625 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.266606    2583 scope.go:117] "RemoveContainer" containerID="4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35"
Jul 28 18:09:30.266919 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:09:30.266896    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35\": container with ID starting with 4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35 not found: ID does not exist" containerID="4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35"
Jul 28 18:09:30.267008 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.266927    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35"} err="failed to get container status \"4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35\": rpc error: code = NotFound desc = could not find container \"4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35\": container with ID starting with 4424f0d0db5da54876f3efe59a9c7b75c73b81bac706d45bf666eb07cf0c6c35 not found: ID does not exist"
Jul 28 18:09:30.267008 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.266944    2583 scope.go:117] "RemoveContainer" containerID="93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd"
Jul 28 18:09:30.267278 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:09:30.267256    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd\": container with ID starting with 93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd not found: ID does not exist" containerID="93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd"
Jul 28 18:09:30.267333 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.267286    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd"} err="failed to get container status \"93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd\": rpc error: code = NotFound desc = could not find container \"93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd\": container with ID starting with 93fbae063f3a0bfb5b4bbebbb2c283c095586dd724ccf48ff07fb1b6f8d389fd not found: ID does not exist"
Jul 28 18:09:30.267333 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.267307    2583 scope.go:117] "RemoveContainer" containerID="e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb"
Jul 28 18:09:30.267804 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:09:30.267630    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb\": container with ID starting with e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb not found: ID does not exist" containerID="e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb"
Jul 28 18:09:30.267959 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.267662    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb"} err="failed to get container status \"e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb\": rpc error: code = NotFound desc = could not find container \"e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb\": container with ID starting with e90dab6cb119173f73baad890e9f3f17c2cda349f0c22baee11f656b12176ffb not found: ID does not exist"
Jul 28 18:09:30.269609 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.269592    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf"]
Jul 28 18:09:30.272920 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:09:30.272900    2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5801bd_e79c_415e_9370_52cf6be18501.slice\": RecentStats: unable to find data in memory cache]"
Jul 28 18:09:30.537108 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:30.537005    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" path="/var/lib/kubelet/pods/ab5801bd-e79c-415e-9370-52cf6be18501/volumes"
Jul 28 18:09:31.042461 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.042425    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:09:31.095808 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.095767    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"]
Jul 28 18:09:31.096190 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.096173    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="storage-initializer"
Jul 28 18:09:31.096190 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.096193    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="storage-initializer"
Jul 28 18:09:31.096357 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.096208    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container"
Jul 28 18:09:31.096357 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.096217    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container"
Jul 28 18:09:31.096357 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.096230    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kube-rbac-proxy"
Jul 28 18:09:31.096357 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.096238    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kube-rbac-proxy"
Jul 28 18:09:31.096357 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.096346    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kserve-container"
Jul 28 18:09:31.096357 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.096358    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5801bd-e79c-415e-9370-52cf6be18501" containerName="kube-rbac-proxy"
Jul 28 18:09:31.101171 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.101148    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.105244 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.105219    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a0e545-predictor-serving-cert\""
Jul 28 18:09:31.105244 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.105236    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a0e545-kube-rbac-proxy-sar-config\""
Jul 28 18:09:31.105512 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.105495    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:09:31.105570 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.105550    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:09:31.105570 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.105559    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:09:31.106160 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.106142    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:09:31.106395 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.106374    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:09:31.112859 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.112835    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"]
Jul 28 18:09:31.202496 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.202462    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.202496 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.202499    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-cabundle-cert\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.202756 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.202532    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kserve-provision-location\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.202756 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.202624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-proxy-tls\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.202756 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.202686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4shlg\" (UniqueName: \"kubernetes.io/projected/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-api-access-4shlg\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.206380 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.206348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:09:31.303478 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.303386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-proxy-tls\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.303478 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.303439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shlg\" (UniqueName: \"kubernetes.io/projected/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-api-access-4shlg\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.303709 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.303481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.303709 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.303501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-cabundle-cert\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.303709 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.303532    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kserve-provision-location\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.304074 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.304051    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kserve-provision-location\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.304303 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.304285    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.304378 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.304344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-cabundle-cert\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.305893 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.305874    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-proxy-tls\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.312590 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.312571    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4shlg\" (UniqueName: \"kubernetes.io/projected/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-api-access-4shlg\") pod \"isvc-primary-a0e545-predictor-865b97b54f-v9n79\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") " pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.410892 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.410833    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:31.542343 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:31.542319    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"]
Jul 28 18:09:31.544592 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:09:31.544565    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed20b91_c0ed_4f5e_80a1_a416bd762a7d.slice/crio-542c04f5606258c9092a53f0352fea205794488d78f09603edd8faf98be2d590 WatchSource:0}: Error finding container 542c04f5606258c9092a53f0352fea205794488d78f09603edd8faf98be2d590: Status 404 returned error can't find the container with id 542c04f5606258c9092a53f0352fea205794488d78f09603edd8faf98be2d590
Jul 28 18:09:32.251859 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:32.251827    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" event={"ID":"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d","Type":"ContainerStarted","Data":"b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba"}
Jul 28 18:09:32.252253 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:32.251892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" event={"ID":"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d","Type":"ContainerStarted","Data":"542c04f5606258c9092a53f0352fea205794488d78f09603edd8faf98be2d590"}
Jul 28 18:09:35.266394 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:35.266358    2583 generic.go:356] "Generic (PLEG): container finished" podID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerID="b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba" exitCode=0
Jul 28 18:09:35.266789 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:35.266432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" event={"ID":"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d","Type":"ContainerDied","Data":"b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba"}
Jul 28 18:09:36.273034 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:36.272987    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" event={"ID":"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d","Type":"ContainerStarted","Data":"35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203"}
Jul 28 18:09:36.273415 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:36.273071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" event={"ID":"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d","Type":"ContainerStarted","Data":"27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b"}
Jul 28 18:09:36.273415 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:36.273278    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:36.297977 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:36.297924    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podStartSLOduration=1.779665104 podStartE2EDuration="5.297908278s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.518243174s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:09:36.29547786 +0000 UTC m=+3142.308857753" watchObservedRunningTime="2026-07-28 18:09:36.297908278 +0000 UTC m=+3142.311288171"
Jul 28 18:09:37.277788 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:37.277758    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:37.279224 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:37.279196    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 18:09:38.281746 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:38.281704    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 18:09:43.286584 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:43.286555    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:09:43.287136 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:43.287108    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 18:09:53.287001 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:09:53.286963    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 18:10:03.287525 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:10:03.287486    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 18:10:13.287229 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:10:13.287188    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 18:10:23.287378 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:10:23.287341    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 18:10:33.287721 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:10:33.287680    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 18:10:43.287921 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:10:43.287883    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 18:10:49.533952 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:10:49.533916    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:11:06.316328 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:06.316241    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:06.452660 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:06.452626    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:20.409770 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:20.409732    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"]
Jul 28 18:11:20.410280 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:20.410049    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container" containerID="cri-o://27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b" gracePeriod=30
Jul 28 18:11:20.410280 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:20.410072    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kube-rbac-proxy" containerID="cri-o://35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203" gracePeriod=30
Jul 28 18:11:20.443782 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:20.443756    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:20.566844 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:20.566814    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:20.718230 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:20.718139    2583 generic.go:356] "Generic (PLEG): container finished" podID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerID="35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203" exitCode=2
Jul 28 18:11:20.718230 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:20.718209    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" event={"ID":"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d","Type":"ContainerDied","Data":"35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203"}
Jul 28 18:11:23.282377 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:23.282331    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 18:11:25.157791 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.157767    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:11:25.171999 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.171976    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kserve-provision-location\") pod \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") "
Jul 28 18:11:25.172125 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.172011    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-cabundle-cert\") pod \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") "
Jul 28 18:11:25.172125 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.172065    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4shlg\" (UniqueName: \"kubernetes.io/projected/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-api-access-4shlg\") pod \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") "
Jul 28 18:11:25.172125 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.172103    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-rbac-proxy-sar-config\") pod \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") "
Jul 28 18:11:25.172259 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.172149    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-proxy-tls\") pod \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\" (UID: \"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d\") "
Jul 28 18:11:25.172329 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.172307    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" (UID: "5ed20b91-c0ed-4f5e-80a1-a416bd762a7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:11:25.172418 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.172387    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" (UID: "5ed20b91-c0ed-4f5e-80a1-a416bd762a7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:11:25.172418 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.172416    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:11:25.172604 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.172513    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" (UID: "5ed20b91-c0ed-4f5e-80a1-a416bd762a7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:11:25.174170 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.174142    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-api-access-4shlg" (OuterVolumeSpecName: "kube-api-access-4shlg") pod "5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" (UID: "5ed20b91-c0ed-4f5e-80a1-a416bd762a7d"). InnerVolumeSpecName "kube-api-access-4shlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:11:25.174253 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.174210    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" (UID: "5ed20b91-c0ed-4f5e-80a1-a416bd762a7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:11:25.273276 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.273250    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:11:25.273276 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.273274    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4shlg\" (UniqueName: \"kubernetes.io/projected/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-api-access-4shlg\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:11:25.273276 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.273283    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:11:25.273526 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.273292    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:11:25.630039 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.629940    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:25.659501 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.659467    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"]
Jul 28 18:11:25.659820 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.659806    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container"
Jul 28 18:11:25.659864 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.659823    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container"
Jul 28 18:11:25.659864 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.659834    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kube-rbac-proxy"
Jul 28 18:11:25.659864 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.659840    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kube-rbac-proxy"
Jul 28 18:11:25.659864 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.659851    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="storage-initializer"
Jul 28 18:11:25.659864 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.659857    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="storage-initializer"
Jul 28 18:11:25.660039 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.659904    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kserve-container"
Jul 28 18:11:25.660039 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.659915    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerName="kube-rbac-proxy"
Jul 28 18:11:25.663289 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.663265    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.666063 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.666037    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-048456-predictor-serving-cert\""
Jul 28 18:11:25.666202 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.666133    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-048456-dockercfg-fdmfp\""
Jul 28 18:11:25.666269 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.666201    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-048456-kube-rbac-proxy-sar-config\""
Jul 28 18:11:25.666269 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.666201    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-048456\""
Jul 28 18:11:25.673417 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.673391    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"]
Jul 28 18:11:25.675976 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.675954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-cabundle-cert\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.676110 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.676003    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90b9cb96-925f-4f5e-95d3-4f759f645b34-proxy-tls\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.676110 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.676047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.676110 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.676076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90b9cb96-925f-4f5e-95d3-4f759f645b34-kserve-provision-location\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.676302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.676123    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvb5\" (UniqueName: \"kubernetes.io/projected/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-api-access-9kvb5\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.740560 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.740529    2583 generic.go:356] "Generic (PLEG): container finished" podID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" containerID="27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b" exitCode=0
Jul 28 18:11:25.740710 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.740589    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"
Jul 28 18:11:25.740710 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.740588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" event={"ID":"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d","Type":"ContainerDied","Data":"27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b"}
Jul 28 18:11:25.740710 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.740657    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79" event={"ID":"5ed20b91-c0ed-4f5e-80a1-a416bd762a7d","Type":"ContainerDied","Data":"542c04f5606258c9092a53f0352fea205794488d78f09603edd8faf98be2d590"}
Jul 28 18:11:25.740710 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.740684    2583 scope.go:117] "RemoveContainer" containerID="35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203"
Jul 28 18:11:25.749703 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.749686    2583 scope.go:117] "RemoveContainer" containerID="27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b"
Jul 28 18:11:25.757271 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.757252    2583 scope.go:117] "RemoveContainer" containerID="b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba"
Jul 28 18:11:25.764117 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.764090    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"]
Jul 28 18:11:25.765367 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.765344    2583 scope.go:117] "RemoveContainer" containerID="35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203"
Jul 28 18:11:25.765733 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:11:25.765709    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203\": container with ID starting with 35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203 not found: ID does not exist" containerID="35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203"
Jul 28 18:11:25.765846 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.765742    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203"} err="failed to get container status \"35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203\": rpc error: code = NotFound desc = could not find container \"35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203\": container with ID starting with 35033a0c50b6d9e0b71c51d5596b3346742b61051fecc1824a8abe35de83e203 not found: ID does not exist"
Jul 28 18:11:25.765846 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.765764    2583 scope.go:117] "RemoveContainer" containerID="27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b"
Jul 28 18:11:25.766109 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:11:25.766071    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b\": container with ID starting with 27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b not found: ID does not exist" containerID="27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b"
Jul 28 18:11:25.766215 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.766108    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b"} err="failed to get container status \"27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b\": rpc error: code = NotFound desc = could not find container \"27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b\": container with ID starting with 27192debc4c84b6e852ca238fcb12c35b301751667e055bd35db62068330ed9b not found: ID does not exist"
Jul 28 18:11:25.766215 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.766133    2583 scope.go:117] "RemoveContainer" containerID="b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba"
Jul 28 18:11:25.766437 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:11:25.766409    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba\": container with ID starting with b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba not found: ID does not exist" containerID="b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba"
Jul 28 18:11:25.766542 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.766441    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba"} err="failed to get container status \"b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba\": rpc error: code = NotFound desc = could not find container \"b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba\": container with ID starting with b47a1e9246b73e6c7757328fa06fab952ab76befbedae314cac8658ae5c06aba not found: ID does not exist"
Jul 28 18:11:25.768284 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.768264    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79"]
Jul 28 18:11:25.777254 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.777233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90b9cb96-925f-4f5e-95d3-4f759f645b34-proxy-tls\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.777343 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.777258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.777343 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.777278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90b9cb96-925f-4f5e-95d3-4f759f645b34-kserve-provision-location\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.777343 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.777304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvb5\" (UniqueName: \"kubernetes.io/projected/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-api-access-9kvb5\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.777505 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.777349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-cabundle-cert\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.777505 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:11:25.777467    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-048456-predictor-serving-cert: secret "isvc-init-fail-048456-predictor-serving-cert" not found
Jul 28 18:11:25.777606 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:11:25.777543    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90b9cb96-925f-4f5e-95d3-4f759f645b34-proxy-tls podName:90b9cb96-925f-4f5e-95d3-4f759f645b34 nodeName:}" failed. No retries permitted until 2026-07-28 18:11:26.277520656 +0000 UTC m=+3252.290900528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/90b9cb96-925f-4f5e-95d3-4f759f645b34-proxy-tls") pod "isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb" (UID: "90b9cb96-925f-4f5e-95d3-4f759f645b34") : secret "isvc-init-fail-048456-predictor-serving-cert" not found
Jul 28 18:11:25.777682 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.777660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90b9cb96-925f-4f5e-95d3-4f759f645b34-kserve-provision-location\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.777914 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.777897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.777914 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.777907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-cabundle-cert\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.786513 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.786494    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvb5\" (UniqueName: \"kubernetes.io/projected/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-api-access-9kvb5\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:25.892159 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:25.892062    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:26.282102 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:26.282066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90b9cb96-925f-4f5e-95d3-4f759f645b34-proxy-tls\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:26.284492 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:26.284470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90b9cb96-925f-4f5e-95d3-4f759f645b34-proxy-tls\") pod \"isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") " pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:26.537648 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:26.537569    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ed20b91-c0ed-4f5e-80a1-a416bd762a7d" path="/var/lib/kubelet/pods/5ed20b91-c0ed-4f5e-80a1-a416bd762a7d/volumes"
Jul 28 18:11:26.574814 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:26.574773    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:26.722593 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:26.722566    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"]
Jul 28 18:11:26.724971 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:11:26.724943    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b9cb96_925f_4f5e_95d3_4f759f645b34.slice/crio-c8d8d79b8112c2de1727c7d259f85a976ec9e3ebb65eb64ff6e5cca9f30a04a7 WatchSource:0}: Error finding container c8d8d79b8112c2de1727c7d259f85a976ec9e3ebb65eb64ff6e5cca9f30a04a7: Status 404 returned error can't find the container with id c8d8d79b8112c2de1727c7d259f85a976ec9e3ebb65eb64ff6e5cca9f30a04a7
Jul 28 18:11:26.746547 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:26.746513    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb" event={"ID":"90b9cb96-925f-4f5e-95d3-4f759f645b34","Type":"ContainerStarted","Data":"c8d8d79b8112c2de1727c7d259f85a976ec9e3ebb65eb64ff6e5cca9f30a04a7"}
Jul 28 18:11:27.753175 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:27.753136    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb" event={"ID":"90b9cb96-925f-4f5e-95d3-4f759f645b34","Type":"ContainerStarted","Data":"f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a"}
Jul 28 18:11:32.775680 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:32.775659    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb_90b9cb96-925f-4f5e-95d3-4f759f645b34/storage-initializer/0.log"
Jul 28 18:11:32.775976 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:32.775695    2583 generic.go:356] "Generic (PLEG): container finished" podID="90b9cb96-925f-4f5e-95d3-4f759f645b34" containerID="f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a" exitCode=1
Jul 28 18:11:32.775976 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:32.775768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb" event={"ID":"90b9cb96-925f-4f5e-95d3-4f759f645b34","Type":"ContainerDied","Data":"f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a"}
Jul 28 18:11:33.781721 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:33.781693    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb_90b9cb96-925f-4f5e-95d3-4f759f645b34/storage-initializer/0.log"
Jul 28 18:11:33.782150 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:33.781799    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb" event={"ID":"90b9cb96-925f-4f5e-95d3-4f759f645b34","Type":"ContainerStarted","Data":"b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71"}
Jul 28 18:11:35.683297 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:35.683263    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"]
Jul 28 18:11:35.683654 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:35.683490    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb" podUID="90b9cb96-925f-4f5e-95d3-4f759f645b34" containerName="storage-initializer" containerID="cri-o://b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71" gracePeriod=30
Jul 28 18:11:36.507893 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.507859    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:36.928595 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.928573    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb_90b9cb96-925f-4f5e-95d3-4f759f645b34/storage-initializer/1.log"
Jul 28 18:11:36.928972 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.928958    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb_90b9cb96-925f-4f5e-95d3-4f759f645b34/storage-initializer/0.log"
Jul 28 18:11:36.931509 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.931494    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:36.972168 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.972136    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kvb5\" (UniqueName: \"kubernetes.io/projected/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-api-access-9kvb5\") pod \"90b9cb96-925f-4f5e-95d3-4f759f645b34\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") "
Jul 28 18:11:36.972356 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.972184    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90b9cb96-925f-4f5e-95d3-4f759f645b34-kserve-provision-location\") pod \"90b9cb96-925f-4f5e-95d3-4f759f645b34\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") "
Jul 28 18:11:36.972356 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.972211    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-cabundle-cert\") pod \"90b9cb96-925f-4f5e-95d3-4f759f645b34\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") "
Jul 28 18:11:36.972356 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.972232    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90b9cb96-925f-4f5e-95d3-4f759f645b34-proxy-tls\") pod \"90b9cb96-925f-4f5e-95d3-4f759f645b34\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") "
Jul 28 18:11:36.972356 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.972276    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-rbac-proxy-sar-config\") pod \"90b9cb96-925f-4f5e-95d3-4f759f645b34\" (UID: \"90b9cb96-925f-4f5e-95d3-4f759f645b34\") "
Jul 28 18:11:36.972588 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.972491    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b9cb96-925f-4f5e-95d3-4f759f645b34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90b9cb96-925f-4f5e-95d3-4f759f645b34" (UID: "90b9cb96-925f-4f5e-95d3-4f759f645b34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:11:36.972653 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.972610    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "90b9cb96-925f-4f5e-95d3-4f759f645b34" (UID: "90b9cb96-925f-4f5e-95d3-4f759f645b34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:11:36.972749 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.972716    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "90b9cb96-925f-4f5e-95d3-4f759f645b34" (UID: "90b9cb96-925f-4f5e-95d3-4f759f645b34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:11:36.974415 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.974390    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-api-access-9kvb5" (OuterVolumeSpecName: "kube-api-access-9kvb5") pod "90b9cb96-925f-4f5e-95d3-4f759f645b34" (UID: "90b9cb96-925f-4f5e-95d3-4f759f645b34"). InnerVolumeSpecName "kube-api-access-9kvb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:11:36.974540 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:36.974475    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b9cb96-925f-4f5e-95d3-4f759f645b34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90b9cb96-925f-4f5e-95d3-4f759f645b34" (UID: "90b9cb96-925f-4f5e-95d3-4f759f645b34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:11:37.073496 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.073409    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kvb5\" (UniqueName: \"kubernetes.io/projected/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-api-access-9kvb5\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:11:37.073496 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.073439    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90b9cb96-925f-4f5e-95d3-4f759f645b34-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:11:37.073496 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.073449    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:11:37.073496 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.073459    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90b9cb96-925f-4f5e-95d3-4f759f645b34-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:11:37.073496 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.073468    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90b9cb96-925f-4f5e-95d3-4f759f645b34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:11:37.800503 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.800470    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb_90b9cb96-925f-4f5e-95d3-4f759f645b34/storage-initializer/1.log"
Jul 28 18:11:37.800891 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.800876    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb_90b9cb96-925f-4f5e-95d3-4f759f645b34/storage-initializer/0.log"
Jul 28 18:11:37.800951 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.800909    2583 generic.go:356] "Generic (PLEG): container finished" podID="90b9cb96-925f-4f5e-95d3-4f759f645b34" containerID="b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71" exitCode=1
Jul 28 18:11:37.800990 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.800942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb" event={"ID":"90b9cb96-925f-4f5e-95d3-4f759f645b34","Type":"ContainerDied","Data":"b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71"}
Jul 28 18:11:37.800990 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.800981    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"
Jul 28 18:11:37.801121 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.801029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb" event={"ID":"90b9cb96-925f-4f5e-95d3-4f759f645b34","Type":"ContainerDied","Data":"c8d8d79b8112c2de1727c7d259f85a976ec9e3ebb65eb64ff6e5cca9f30a04a7"}
Jul 28 18:11:37.801121 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.801058    2583 scope.go:117] "RemoveContainer" containerID="b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71"
Jul 28 18:11:37.810236 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.810216    2583 scope.go:117] "RemoveContainer" containerID="f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a"
Jul 28 18:11:37.817853 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.817837    2583 scope.go:117] "RemoveContainer" containerID="b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71"
Jul 28 18:11:37.818106 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:11:37.818088    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71\": container with ID starting with b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71 not found: ID does not exist" containerID="b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71"
Jul 28 18:11:37.818151 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.818115    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71"} err="failed to get container status \"b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71\": rpc error: code = NotFound desc = could not find container \"b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71\": container with ID starting with b316eaa4adf0c13ef185d217ccf03205ea9dce443abdd36ea77adae9218a2c71 not found: ID does not exist"
Jul 28 18:11:37.818151 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.818133    2583 scope.go:117] "RemoveContainer" containerID="f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a"
Jul 28 18:11:37.818357 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:11:37.818342    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a\": container with ID starting with f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a not found: ID does not exist" containerID="f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a"
Jul 28 18:11:37.818404 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.818361    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a"} err="failed to get container status \"f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a\": rpc error: code = NotFound desc = could not find container \"f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a\": container with ID starting with f6cecb2da165fa6042f2593c9be957a425e62603b1dfbbd7e7f00f6c26439d6a not found: ID does not exist"
Jul 28 18:11:37.841835 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.841799    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"]
Jul 28 18:11:37.849209 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:37.849182    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb"]
Jul 28 18:11:38.537160 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:38.537119    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b9cb96-925f-4f5e-95d3-4f759f645b34" path="/var/lib/kubelet/pods/90b9cb96-925f-4f5e-95d3-4f759f645b34/volumes"
Jul 28 18:11:40.921587 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:40.921550    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:41.066269 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:11:41.066234    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:12:23.089360 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:12:23.089242    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:12:23.092932 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:12:23.092914    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:13:31.888748 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:31.888711    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:36.262995 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.262962    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:36.316046 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.315991    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"]
Jul 28 18:13:36.316443 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.316424    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b9cb96-925f-4f5e-95d3-4f759f645b34" containerName="storage-initializer"
Jul 28 18:13:36.316539 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.316446    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b9cb96-925f-4f5e-95d3-4f759f645b34" containerName="storage-initializer"
Jul 28 18:13:36.316598 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.316552    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b9cb96-925f-4f5e-95d3-4f759f645b34" containerName="storage-initializer"
Jul 28 18:13:36.316598 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.316566    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b9cb96-925f-4f5e-95d3-4f759f645b34" containerName="storage-initializer"
Jul 28 18:13:36.316765 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.316646    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b9cb96-925f-4f5e-95d3-4f759f645b34" containerName="storage-initializer"
Jul 28 18:13:36.316765 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.316656    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b9cb96-925f-4f5e-95d3-4f759f645b34" containerName="storage-initializer"
Jul 28 18:13:36.319750 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.319729    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.322898 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.322878    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:13:36.323503 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.323483    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:13:36.323584 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.323545    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:13:36.323741 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.323728    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:13:36.323741 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.323734    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:13:36.323861 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.323759    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 18:13:36.323861 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.323761    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 28 18:13:36.334114 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.334094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"]
Jul 28 18:13:36.379040 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.378966    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:36.387121 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.387097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln95b\" (UniqueName: \"kubernetes.io/projected/7e38f478-a703-440a-acef-f958ff518a93-kube-api-access-ln95b\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.387246 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.387133    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e38f478-a703-440a-acef-f958ff518a93-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.387246 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.387157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e38f478-a703-440a-acef-f958ff518a93-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.387246 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.387234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.387380 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.387310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.487992 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.487948    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.488207 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.488058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.488207 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.488082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln95b\" (UniqueName: \"kubernetes.io/projected/7e38f478-a703-440a-acef-f958ff518a93-kube-api-access-ln95b\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.488207 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.488108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e38f478-a703-440a-acef-f958ff518a93-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.488207 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.488144    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e38f478-a703-440a-acef-f958ff518a93-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.488513 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.488492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e38f478-a703-440a-acef-f958ff518a93-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.488671 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.488651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.488742 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.488725    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.490504 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.490484    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e38f478-a703-440a-acef-f958ff518a93-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.498635 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.498614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln95b\" (UniqueName: \"kubernetes.io/projected/7e38f478-a703-440a-acef-f958ff518a93-kube-api-access-ln95b\") pod \"isvc-predictive-xgboost-predictor-695f978978-z2vth\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.630511 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.630416    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:13:36.762345 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:36.762321    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"]
Jul 28 18:13:36.764838 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:13:36.764810    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e38f478_a703_440a_acef_f958ff518a93.slice/crio-203d05118f34d78890a0ce570f193b1c94a9c9ea0eb945b22ee92af3e5fc802f WatchSource:0}: Error finding container 203d05118f34d78890a0ce570f193b1c94a9c9ea0eb945b22ee92af3e5fc802f: Status 404 returned error can't find the container with id 203d05118f34d78890a0ce570f193b1c94a9c9ea0eb945b22ee92af3e5fc802f
Jul 28 18:13:37.315247 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:37.315214    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" event={"ID":"7e38f478-a703-440a-acef-f958ff518a93","Type":"ContainerStarted","Data":"085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d"}
Jul 28 18:13:37.315680 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:37.315272    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" event={"ID":"7e38f478-a703-440a-acef-f958ff518a93","Type":"ContainerStarted","Data":"203d05118f34d78890a0ce570f193b1c94a9c9ea0eb945b22ee92af3e5fc802f"}
Jul 28 18:13:39.016964 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:39.016921    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:41.333298 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:41.333260    2583 generic.go:356] "Generic (PLEG): container finished" podID="7e38f478-a703-440a-acef-f958ff518a93" containerID="085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d" exitCode=0
Jul 28 18:13:41.333715 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:41.333336    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" event={"ID":"7e38f478-a703-440a-acef-f958ff518a93","Type":"ContainerDied","Data":"085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d"}
Jul 28 18:13:41.334638 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:13:41.334619    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:14:03.456431 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:03.456347    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" event={"ID":"7e38f478-a703-440a-acef-f958ff518a93","Type":"ContainerStarted","Data":"be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486"}
Jul 28 18:14:03.456431 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:03.456411    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" event={"ID":"7e38f478-a703-440a-acef-f958ff518a93","Type":"ContainerStarted","Data":"72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b"}
Jul 28 18:14:03.456957 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:03.456599    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:14:03.485192 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:03.485126    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podStartSLOduration=1.996763302 podStartE2EDuration="27.485109546s" totalImagesPullingTime="21.644558264s" totalInitContainerRuntime="3.84378798s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:14:03.484204585 +0000 UTC m=+3409.497584502" watchObservedRunningTime="2026-07-28 18:14:03.485109546 +0000 UTC m=+3409.498489440"
Jul 28 18:14:04.460229 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:04.460194    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:14:04.461725 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:04.461697    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 18:14:05.464349 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:05.464315    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 18:14:10.469072 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:10.469014    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:14:10.469538 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:10.469481    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 18:14:14.234957 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:14.234916    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:14:20.470374 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:20.470321    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 18:14:30.469982 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:30.469943    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 18:14:40.253256 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:40.253207    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:14:40.470200 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:40.470161    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 18:14:50.470397 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:14:50.470352    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 18:15:00.470565 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:00.470521    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 18:15:10.469710 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:10.469673    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 18:15:20.470662 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:20.470629    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:15:26.396003 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:26.395963    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"]
Jul 28 18:15:26.396482 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:26.396261    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container" containerID="cri-o://72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b" gracePeriod=30
Jul 28 18:15:26.396482 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:26.396278    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kube-rbac-proxy" containerID="cri-o://be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486" gracePeriod=30
Jul 28 18:15:26.429308 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:26.429277    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:15:26.816775 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:26.816742    2583 generic.go:356] "Generic (PLEG): container finished" podID="7e38f478-a703-440a-acef-f958ff518a93" containerID="be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486" exitCode=2
Jul 28 18:15:26.816963 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:26.816819    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" event={"ID":"7e38f478-a703-440a-acef-f958ff518a93","Type":"ContainerDied","Data":"be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486"}
Jul 28 18:15:30.465639 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:30.465597    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 18:15:30.469951 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:30.469919    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 18:15:31.239137 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.239109    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:15:31.395984 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.395904    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e38f478-a703-440a-acef-f958ff518a93-kserve-provision-location\") pod \"7e38f478-a703-440a-acef-f958ff518a93\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") "
Jul 28 18:15:31.395984 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.395944    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-kube-rbac-proxy-sar-config\") pod \"7e38f478-a703-440a-acef-f958ff518a93\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") "
Jul 28 18:15:31.395984 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.395963    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-cabundle-cert\") pod \"7e38f478-a703-440a-acef-f958ff518a93\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") "
Jul 28 18:15:31.396305 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.396056    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln95b\" (UniqueName: \"kubernetes.io/projected/7e38f478-a703-440a-acef-f958ff518a93-kube-api-access-ln95b\") pod \"7e38f478-a703-440a-acef-f958ff518a93\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") "
Jul 28 18:15:31.396305 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.396074    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e38f478-a703-440a-acef-f958ff518a93-proxy-tls\") pod \"7e38f478-a703-440a-acef-f958ff518a93\" (UID: \"7e38f478-a703-440a-acef-f958ff518a93\") "
Jul 28 18:15:31.396305 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.396280    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e38f478-a703-440a-acef-f958ff518a93-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e38f478-a703-440a-acef-f958ff518a93" (UID: "7e38f478-a703-440a-acef-f958ff518a93"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:15:31.396452 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.396326    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e38f478-a703-440a-acef-f958ff518a93" (UID: "7e38f478-a703-440a-acef-f958ff518a93"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:15:31.396452 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.396360    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e38f478-a703-440a-acef-f958ff518a93" (UID: "7e38f478-a703-440a-acef-f958ff518a93"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:15:31.398171 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.398143    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e38f478-a703-440a-acef-f958ff518a93-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e38f478-a703-440a-acef-f958ff518a93" (UID: "7e38f478-a703-440a-acef-f958ff518a93"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:15:31.398294 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.398238    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e38f478-a703-440a-acef-f958ff518a93-kube-api-access-ln95b" (OuterVolumeSpecName: "kube-api-access-ln95b") pod "7e38f478-a703-440a-acef-f958ff518a93" (UID: "7e38f478-a703-440a-acef-f958ff518a93"). InnerVolumeSpecName "kube-api-access-ln95b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:15:31.497167 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.497129    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e38f478-a703-440a-acef-f958ff518a93-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:15:31.497167 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.497159    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:15:31.497167 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.497168    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e38f478-a703-440a-acef-f958ff518a93-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:15:31.497167 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.497177    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln95b\" (UniqueName: \"kubernetes.io/projected/7e38f478-a703-440a-acef-f958ff518a93-kube-api-access-ln95b\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:15:31.497642 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.497186    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e38f478-a703-440a-acef-f958ff518a93-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:15:31.521234 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.521202    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:15:31.600786 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.600752    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"]
Jul 28 18:15:31.601187 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.601164    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="storage-initializer"
Jul 28 18:15:31.601187 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.601180    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="storage-initializer"
Jul 28 18:15:31.601187 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.601190    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kube-rbac-proxy"
Jul 28 18:15:31.601337 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.601196    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kube-rbac-proxy"
Jul 28 18:15:31.601337 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.601207    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container"
Jul 28 18:15:31.601337 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.601213    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container"
Jul 28 18:15:31.601337 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.601281    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kube-rbac-proxy"
Jul 28 18:15:31.601337 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.601290    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e38f478-a703-440a-acef-f958ff518a93" containerName="kserve-container"
Jul 28 18:15:31.603637 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.603621    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.606059 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.606031    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 28 18:15:31.606161 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.606035    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 18:15:31.616613 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.616590    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"]
Jul 28 18:15:31.698693 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.698608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.698693 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.698664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b809cd57-48a3-498a-b829-9144b37e8033-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.698911 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.698827    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b809cd57-48a3-498a-b829-9144b37e8033-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.698981 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.698949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.699071 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.698998    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtz5r\" (UniqueName: \"kubernetes.io/projected/b809cd57-48a3-498a-b829-9144b37e8033-kube-api-access-rtz5r\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.762305 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.762269    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:15:31.799643 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.799607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.799643 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.799647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b809cd57-48a3-498a-b829-9144b37e8033-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.799895 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.799693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b809cd57-48a3-498a-b829-9144b37e8033-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.799895 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.799720    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.799895 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.799738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtz5r\" (UniqueName: \"kubernetes.io/projected/b809cd57-48a3-498a-b829-9144b37e8033-kube-api-access-rtz5r\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.800352 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.800320    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b809cd57-48a3-498a-b829-9144b37e8033-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.800514 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.800489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.800514 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.800506    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.802273 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.802253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b809cd57-48a3-498a-b829-9144b37e8033-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.818601 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.818569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtz5r\" (UniqueName: \"kubernetes.io/projected/b809cd57-48a3-498a-b829-9144b37e8033-kube-api-access-rtz5r\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:31.840238 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.840199    2583 generic.go:356] "Generic (PLEG): container finished" podID="7e38f478-a703-440a-acef-f958ff518a93" containerID="72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b" exitCode=0
Jul 28 18:15:31.840383 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.840252    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"
Jul 28 18:15:31.840383 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.840278    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" event={"ID":"7e38f478-a703-440a-acef-f958ff518a93","Type":"ContainerDied","Data":"72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b"}
Jul 28 18:15:31.840383 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.840335    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth" event={"ID":"7e38f478-a703-440a-acef-f958ff518a93","Type":"ContainerDied","Data":"203d05118f34d78890a0ce570f193b1c94a9c9ea0eb945b22ee92af3e5fc802f"}
Jul 28 18:15:31.840383 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.840361    2583 scope.go:117] "RemoveContainer" containerID="be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486"
Jul 28 18:15:31.848955 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.848935    2583 scope.go:117] "RemoveContainer" containerID="72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b"
Jul 28 18:15:31.856398 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.856379    2583 scope.go:117] "RemoveContainer" containerID="085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d"
Jul 28 18:15:31.864010 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.863991    2583 scope.go:117] "RemoveContainer" containerID="be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486"
Jul 28 18:15:31.864313 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:15:31.864294    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486\": container with ID starting with be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486 not found: ID does not exist" containerID="be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486"
Jul 28 18:15:31.864375 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.864322    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486"} err="failed to get container status \"be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486\": rpc error: code = NotFound desc = could not find container \"be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486\": container with ID starting with be5c33748a2400e3ad60f949601b200e5959e8d493640b94753a5ff07e4da486 not found: ID does not exist"
Jul 28 18:15:31.864375 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.864341    2583 scope.go:117] "RemoveContainer" containerID="72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b"
Jul 28 18:15:31.864603 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:15:31.864585    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b\": container with ID starting with 72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b not found: ID does not exist" containerID="72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b"
Jul 28 18:15:31.864667 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.864613    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b"} err="failed to get container status \"72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b\": rpc error: code = NotFound desc = could not find container \"72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b\": container with ID starting with 72609694dce85400729fd7599e8bd8f63914489c5e347c66806f3436b7ecc12b not found: ID does not exist"
Jul 28 18:15:31.864667 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.864636    2583 scope.go:117] "RemoveContainer" containerID="085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d"
Jul 28 18:15:31.864901 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:15:31.864883    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d\": container with ID starting with 085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d not found: ID does not exist" containerID="085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d"
Jul 28 18:15:31.864940 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.864911    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d"} err="failed to get container status \"085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d\": rpc error: code = NotFound desc = could not find container \"085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d\": container with ID starting with 085a237f70a019d7fac4e1a542a0d206ae3b36a576f04948e8dbe2a03324e52d not found: ID does not exist"
Jul 28 18:15:31.867839 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.867816    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"]
Jul 28 18:15:31.876600 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.876579    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth"]
Jul 28 18:15:31.917129 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:31.917084    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:32.061331 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:32.061296    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"]
Jul 28 18:15:32.064457 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:15:32.064429    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb809cd57_48a3_498a_b829_9144b37e8033.slice/crio-3ab5b0ff2af23a4b2257585b2a02a234a8c5995af886ff4381e00548847e8615 WatchSource:0}: Error finding container 3ab5b0ff2af23a4b2257585b2a02a234a8c5995af886ff4381e00548847e8615: Status 404 returned error can't find the container with id 3ab5b0ff2af23a4b2257585b2a02a234a8c5995af886ff4381e00548847e8615
Jul 28 18:15:32.537131 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:32.537092    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e38f478-a703-440a-acef-f958ff518a93" path="/var/lib/kubelet/pods/7e38f478-a703-440a-acef-f958ff518a93/volumes"
Jul 28 18:15:32.848339 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:32.848238    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" event={"ID":"b809cd57-48a3-498a-b829-9144b37e8033","Type":"ContainerStarted","Data":"1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972"}
Jul 28 18:15:32.848339 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:32.848313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" event={"ID":"b809cd57-48a3-498a-b829-9144b37e8033","Type":"ContainerStarted","Data":"3ab5b0ff2af23a4b2257585b2a02a234a8c5995af886ff4381e00548847e8615"}
Jul 28 18:15:36.867300 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:36.867269    2583 generic.go:356] "Generic (PLEG): container finished" podID="b809cd57-48a3-498a-b829-9144b37e8033" containerID="1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972" exitCode=0
Jul 28 18:15:36.867699 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:36.867348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" event={"ID":"b809cd57-48a3-498a-b829-9144b37e8033","Type":"ContainerDied","Data":"1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972"}
Jul 28 18:15:37.873338 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:37.873302    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" event={"ID":"b809cd57-48a3-498a-b829-9144b37e8033","Type":"ContainerStarted","Data":"dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5"}
Jul 28 18:15:37.873693 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:37.873366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" event={"ID":"b809cd57-48a3-498a-b829-9144b37e8033","Type":"ContainerStarted","Data":"039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f"}
Jul 28 18:15:37.873693 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:37.873631    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:37.873823 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:37.873758    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:37.875128 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:37.875101    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 18:15:37.906138 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:37.906086    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podStartSLOduration=3.141083131 podStartE2EDuration="6.906069456s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.764986325s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:15:37.903328833 +0000 UTC m=+3503.916708725" watchObservedRunningTime="2026-07-28 18:15:37.906069456 +0000 UTC m=+3503.919449361"
Jul 28 18:15:38.877355 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:38.877317    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 18:15:43.881699 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:43.881669    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:15:43.882335 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:43.882310    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 18:15:53.882754 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:15:53.882710    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 18:16:03.882729 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:16:03.882684    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 18:16:13.882919 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:16:13.882873    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 18:16:23.882817 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:16:23.882777    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 18:16:33.883147 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:16:33.883103    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 18:16:43.883015 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:16:43.882975    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 18:16:53.533489 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:16:53.533457    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:17:01.688704 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:01.688670    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"]
Jul 28 18:17:01.689483 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:01.688881    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container" containerID="cri-o://039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f" gracePeriod=30
Jul 28 18:17:01.689483 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:01.688952    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kube-rbac-proxy" containerID="cri-o://dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5" gracePeriod=30
Jul 28 18:17:01.704579 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:01.704559    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:17:02.244521 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:02.244488    2583 generic.go:356] "Generic (PLEG): container finished" podID="b809cd57-48a3-498a-b829-9144b37e8033" containerID="dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5" exitCode=2
Jul 28 18:17:02.244695 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:02.244556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" event={"ID":"b809cd57-48a3-498a-b829-9144b37e8033","Type":"ContainerDied","Data":"dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5"}
Jul 28 18:17:03.532398 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:03.532353    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 18:17:03.877853 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:03.877761    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 18:17:06.628249 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.628221    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:17:06.690546 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.690453    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtz5r\" (UniqueName: \"kubernetes.io/projected/b809cd57-48a3-498a-b829-9144b37e8033-kube-api-access-rtz5r\") pod \"b809cd57-48a3-498a-b829-9144b37e8033\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") "
Jul 28 18:17:06.690546 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.690498    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-cabundle-cert\") pod \"b809cd57-48a3-498a-b829-9144b37e8033\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") "
Jul 28 18:17:06.690777 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.690562    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b809cd57-48a3-498a-b829-9144b37e8033-kserve-provision-location\") pod \"b809cd57-48a3-498a-b829-9144b37e8033\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") "
Jul 28 18:17:06.690777 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.690588    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b809cd57-48a3-498a-b829-9144b37e8033-proxy-tls\") pod \"b809cd57-48a3-498a-b829-9144b37e8033\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") "
Jul 28 18:17:06.690777 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.690699    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-kube-rbac-proxy-sar-config\") pod \"b809cd57-48a3-498a-b829-9144b37e8033\" (UID: \"b809cd57-48a3-498a-b829-9144b37e8033\") "
Jul 28 18:17:06.691006 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.690974    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b809cd57-48a3-498a-b829-9144b37e8033-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b809cd57-48a3-498a-b829-9144b37e8033" (UID: "b809cd57-48a3-498a-b829-9144b37e8033"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:17:06.691140 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.691014    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b809cd57-48a3-498a-b829-9144b37e8033" (UID: "b809cd57-48a3-498a-b829-9144b37e8033"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:17:06.691140 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.691064    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b809cd57-48a3-498a-b829-9144b37e8033" (UID: "b809cd57-48a3-498a-b829-9144b37e8033"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:17:06.692708 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.692685    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b809cd57-48a3-498a-b829-9144b37e8033-kube-api-access-rtz5r" (OuterVolumeSpecName: "kube-api-access-rtz5r") pod "b809cd57-48a3-498a-b829-9144b37e8033" (UID: "b809cd57-48a3-498a-b829-9144b37e8033"). InnerVolumeSpecName "kube-api-access-rtz5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:17:06.692801 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.692737    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b809cd57-48a3-498a-b829-9144b37e8033-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b809cd57-48a3-498a-b829-9144b37e8033" (UID: "b809cd57-48a3-498a-b829-9144b37e8033"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:17:06.791412 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.791376    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:17:06.791412 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.791407    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtz5r\" (UniqueName: \"kubernetes.io/projected/b809cd57-48a3-498a-b829-9144b37e8033-kube-api-access-rtz5r\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:17:06.791412 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.791418    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b809cd57-48a3-498a-b829-9144b37e8033-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:17:06.791643 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.791426    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b809cd57-48a3-498a-b829-9144b37e8033-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:17:06.791643 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.791435    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b809cd57-48a3-498a-b829-9144b37e8033-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:17:06.820213 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.820165    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:17:06.921820 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.921787    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"]
Jul 28 18:17:06.922128 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.922114    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container"
Jul 28 18:17:06.922179 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.922130    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container"
Jul 28 18:17:06.922179 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.922143    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="storage-initializer"
Jul 28 18:17:06.922179 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.922149    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="storage-initializer"
Jul 28 18:17:06.922179 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.922164    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kube-rbac-proxy"
Jul 28 18:17:06.922179 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.922170    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kube-rbac-proxy"
Jul 28 18:17:06.922333 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.922220    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kube-rbac-proxy"
Jul 28 18:17:06.922333 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.922231    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b809cd57-48a3-498a-b829-9144b37e8033" containerName="kserve-container"
Jul 28 18:17:06.925481 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.925459    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:06.928560 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.928535    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 28 18:17:06.928878 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.928856    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:17:06.940259 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.940226    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"]
Jul 28 18:17:06.993583 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.993544    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba0ce12-9678-44c0-a520-89e628d76e4e-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:06.993758 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.993598    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba0ce12-9678-44c0-a520-89e628d76e4e-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:06.993758 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.993629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm2fq\" (UniqueName: \"kubernetes.io/projected/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-api-access-tm2fq\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:06.993758 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.993651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:06.993758 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:06.993667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.071904 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.071875    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:17:07.095227 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.095196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba0ce12-9678-44c0-a520-89e628d76e4e-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.095374 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.095241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba0ce12-9678-44c0-a520-89e628d76e4e-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.095374 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.095269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm2fq\" (UniqueName: \"kubernetes.io/projected/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-api-access-tm2fq\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.095374 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.095299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.095374 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.095322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.095671 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.095650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba0ce12-9678-44c0-a520-89e628d76e4e-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.095994 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.095974    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.096067 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.095978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.097660 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.097640    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba0ce12-9678-44c0-a520-89e628d76e4e-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.104175 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.104154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm2fq\" (UniqueName: \"kubernetes.io/projected/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-api-access-tm2fq\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.236278 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.236241    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:07.268160 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.268132    2583 generic.go:356] "Generic (PLEG): container finished" podID="b809cd57-48a3-498a-b829-9144b37e8033" containerID="039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f" exitCode=0
Jul 28 18:17:07.268326 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.268169    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" event={"ID":"b809cd57-48a3-498a-b829-9144b37e8033","Type":"ContainerDied","Data":"039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f"}
Jul 28 18:17:07.268326 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.268191    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"
Jul 28 18:17:07.268326 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.268218    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h" event={"ID":"b809cd57-48a3-498a-b829-9144b37e8033","Type":"ContainerDied","Data":"3ab5b0ff2af23a4b2257585b2a02a234a8c5995af886ff4381e00548847e8615"}
Jul 28 18:17:07.268326 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.268246    2583 scope.go:117] "RemoveContainer" containerID="dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5"
Jul 28 18:17:07.277745 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.277722    2583 scope.go:117] "RemoveContainer" containerID="039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f"
Jul 28 18:17:07.285990 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.285949    2583 scope.go:117] "RemoveContainer" containerID="1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972"
Jul 28 18:17:07.296974 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.296928    2583 scope.go:117] "RemoveContainer" containerID="dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5"
Jul 28 18:17:07.297801 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:17:07.297777    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5\": container with ID starting with dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5 not found: ID does not exist" containerID="dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5"
Jul 28 18:17:07.297883 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.297809    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5"} err="failed to get container status \"dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5\": rpc error: code = NotFound desc = could not find container \"dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5\": container with ID starting with dff39b860df59eec665e34362155bcf1f08083d3ad0ab8c3302ca26c2ae6d0f5 not found: ID does not exist"
Jul 28 18:17:07.297883 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.297827    2583 scope.go:117] "RemoveContainer" containerID="039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f"
Jul 28 18:17:07.298167 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:17:07.298137    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f\": container with ID starting with 039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f not found: ID does not exist" containerID="039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f"
Jul 28 18:17:07.298242 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.298178    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f"} err="failed to get container status \"039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f\": rpc error: code = NotFound desc = could not find container \"039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f\": container with ID starting with 039145ad2bf747616c64620e39617a84a7b98e694fe478f47aaa9e6a272c668f not found: ID does not exist"
Jul 28 18:17:07.298242 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.298207    2583 scope.go:117] "RemoveContainer" containerID="1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972"
Jul 28 18:17:07.298526 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:17:07.298503    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972\": container with ID starting with 1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972 not found: ID does not exist" containerID="1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972"
Jul 28 18:17:07.298580 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.298538    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972"} err="failed to get container status \"1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972\": rpc error: code = NotFound desc = could not find container \"1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972\": container with ID starting with 1532b0e5ca323887b09492afac8095865ab61fd1fd654962dde7157c184a4972 not found: ID does not exist"
Jul 28 18:17:07.307397 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.307371    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"]
Jul 28 18:17:07.311694 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.311672    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h"]
Jul 28 18:17:07.378784 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:07.378759    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"]
Jul 28 18:17:07.381289 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:17:07.381267    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba0ce12_9678_44c0_a520_89e628d76e4e.slice/crio-c3a00c922eb41fd5be96aab4263f3adf21703b1c46d2120ef4940a1046e4544f WatchSource:0}: Error finding container c3a00c922eb41fd5be96aab4263f3adf21703b1c46d2120ef4940a1046e4544f: Status 404 returned error can't find the container with id c3a00c922eb41fd5be96aab4263f3adf21703b1c46d2120ef4940a1046e4544f
Jul 28 18:17:08.274739 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:08.274704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" event={"ID":"dba0ce12-9678-44c0-a520-89e628d76e4e","Type":"ContainerStarted","Data":"1ea80d7bb96b678afd5bbdca74626b2f2b62b6abc1e46c6428927398317f63bb"}
Jul 28 18:17:08.275164 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:08.274759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" event={"ID":"dba0ce12-9678-44c0-a520-89e628d76e4e","Type":"ContainerStarted","Data":"c3a00c922eb41fd5be96aab4263f3adf21703b1c46d2120ef4940a1046e4544f"}
Jul 28 18:17:08.538509 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:08.538404    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b809cd57-48a3-498a-b829-9144b37e8033" path="/var/lib/kubelet/pods/b809cd57-48a3-498a-b829-9144b37e8033/volumes"
Jul 28 18:17:12.293760 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:12.293668    2583 generic.go:356] "Generic (PLEG): container finished" podID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerID="1ea80d7bb96b678afd5bbdca74626b2f2b62b6abc1e46c6428927398317f63bb" exitCode=0
Jul 28 18:17:12.294134 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:12.293748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" event={"ID":"dba0ce12-9678-44c0-a520-89e628d76e4e","Type":"ContainerDied","Data":"1ea80d7bb96b678afd5bbdca74626b2f2b62b6abc1e46c6428927398317f63bb"}
Jul 28 18:17:13.300592 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:13.300557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" event={"ID":"dba0ce12-9678-44c0-a520-89e628d76e4e","Type":"ContainerStarted","Data":"e3710338bd900ade8a06dbc0c5a3a78d842c0fa64a46993166796d33251bf993"}
Jul 28 18:17:13.301080 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:13.300627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" event={"ID":"dba0ce12-9678-44c0-a520-89e628d76e4e","Type":"ContainerStarted","Data":"ee0c27fbc58846cc0a565730aaf5cee2de1d7d5e3113940f93158e69c3575503"}
Jul 28 18:17:13.301080 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:13.300852    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:13.321601 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:13.321550    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" podStartSLOduration=3.350114536 podStartE2EDuration="7.321536551s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.971422015s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:17:13.320235106 +0000 UTC m=+3599.333614990" watchObservedRunningTime="2026-07-28 18:17:13.321536551 +0000 UTC m=+3599.334916443"
Jul 28 18:17:14.304631 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:14.304600    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:20.314947 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:20.314920    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:17:23.116294 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:23.116168    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:17:23.137091 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:23.120351    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:17:50.316685 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:17:50.316641    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 18:18:00.315916 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:00.315873    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 18:18:10.315956 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:10.315914    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 18:18:20.315578 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:20.315538    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 18:18:28.642613 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:28.642584    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:18:35.347759 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:35.347718    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:36.731258 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:36.731228    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:37.852959 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:37.852928    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"]
Jul 28 18:18:37.853422 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:37.853206    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kserve-container" containerID="cri-o://ee0c27fbc58846cc0a565730aaf5cee2de1d7d5e3113940f93158e69c3575503" gracePeriod=30
Jul 28 18:18:37.853422 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:37.853263    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kube-rbac-proxy" containerID="cri-o://e3710338bd900ade8a06dbc0c5a3a78d842c0fa64a46993166796d33251bf993" gracePeriod=30
Jul 28 18:18:37.869253 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:37.869230    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:38.532536 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:38.532490    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 18:18:38.682792 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:38.682761    2583 generic.go:356] "Generic (PLEG): container finished" podID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerID="e3710338bd900ade8a06dbc0c5a3a78d842c0fa64a46993166796d33251bf993" exitCode=2
Jul 28 18:18:38.682938 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:38.682833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" event={"ID":"dba0ce12-9678-44c0-a520-89e628d76e4e","Type":"ContainerDied","Data":"e3710338bd900ade8a06dbc0c5a3a78d842c0fa64a46993166796d33251bf993"}
Jul 28 18:18:40.310524 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:40.310483    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 18:18:42.701480 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.701450    2583 generic.go:356] "Generic (PLEG): container finished" podID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerID="ee0c27fbc58846cc0a565730aaf5cee2de1d7d5e3113940f93158e69c3575503" exitCode=0
Jul 28 18:18:42.701786 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.701518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" event={"ID":"dba0ce12-9678-44c0-a520-89e628d76e4e","Type":"ContainerDied","Data":"ee0c27fbc58846cc0a565730aaf5cee2de1d7d5e3113940f93158e69c3575503"}
Jul 28 18:18:42.701786 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.701574    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd" event={"ID":"dba0ce12-9678-44c0-a520-89e628d76e4e","Type":"ContainerDied","Data":"c3a00c922eb41fd5be96aab4263f3adf21703b1c46d2120ef4940a1046e4544f"}
Jul 28 18:18:42.701786 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.701596    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3a00c922eb41fd5be96aab4263f3adf21703b1c46d2120ef4940a1046e4544f"
Jul 28 18:18:42.702125 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.702107    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:18:42.818461 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.818372    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba0ce12-9678-44c0-a520-89e628d76e4e-kserve-provision-location\") pod \"dba0ce12-9678-44c0-a520-89e628d76e4e\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") "
Jul 28 18:18:42.818461 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.818415    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-cabundle-cert\") pod \"dba0ce12-9678-44c0-a520-89e628d76e4e\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") "
Jul 28 18:18:42.818461 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.818456    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba0ce12-9678-44c0-a520-89e628d76e4e-proxy-tls\") pod \"dba0ce12-9678-44c0-a520-89e628d76e4e\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") "
Jul 28 18:18:42.818750 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.818472    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm2fq\" (UniqueName: \"kubernetes.io/projected/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-api-access-tm2fq\") pod \"dba0ce12-9678-44c0-a520-89e628d76e4e\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") "
Jul 28 18:18:42.818750 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.818492    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-rbac-proxy-sar-config\") pod \"dba0ce12-9678-44c0-a520-89e628d76e4e\" (UID: \"dba0ce12-9678-44c0-a520-89e628d76e4e\") "
Jul 28 18:18:42.818853 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.818751    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba0ce12-9678-44c0-a520-89e628d76e4e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dba0ce12-9678-44c0-a520-89e628d76e4e" (UID: "dba0ce12-9678-44c0-a520-89e628d76e4e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:18:42.818894 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.818871    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dba0ce12-9678-44c0-a520-89e628d76e4e" (UID: "dba0ce12-9678-44c0-a520-89e628d76e4e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:18:42.818926 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.818877    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dba0ce12-9678-44c0-a520-89e628d76e4e" (UID: "dba0ce12-9678-44c0-a520-89e628d76e4e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:18:42.820601 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.820576    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba0ce12-9678-44c0-a520-89e628d76e4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dba0ce12-9678-44c0-a520-89e628d76e4e" (UID: "dba0ce12-9678-44c0-a520-89e628d76e4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:18:42.820804 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.820791    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-api-access-tm2fq" (OuterVolumeSpecName: "kube-api-access-tm2fq") pod "dba0ce12-9678-44c0-a520-89e628d76e4e" (UID: "dba0ce12-9678-44c0-a520-89e628d76e4e"). InnerVolumeSpecName "kube-api-access-tm2fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:18:42.919775 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.919743    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:18:42.919775 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.919768    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba0ce12-9678-44c0-a520-89e628d76e4e-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:18:42.919775 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.919776    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm2fq\" (UniqueName: \"kubernetes.io/projected/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-api-access-tm2fq\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:18:42.919995 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.919785    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba0ce12-9678-44c0-a520-89e628d76e4e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:18:42.919995 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.919794    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dba0ce12-9678-44c0-a520-89e628d76e4e-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:18:42.936000 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:42.935970    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:43.212110 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:43.212006    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:43.706076 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:43.706045    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"
Jul 28 18:18:43.727756 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:43.727726    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"]
Jul 28 18:18:43.731847 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:43.731822    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd"]
Jul 28 18:18:44.537584 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:18:44.537543    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" path="/var/lib/kubelet/pods/dba0ce12-9678-44c0-a520-89e628d76e4e/volumes"
Jul 28 18:20:13.571167 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:13.571081    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:20:18.243070 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.243010    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:20:18.291670 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.291633    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"]
Jul 28 18:20:18.291985 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.291973    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kserve-container"
Jul 28 18:20:18.292064 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.291986    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kserve-container"
Jul 28 18:20:18.292064 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.291997    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kube-rbac-proxy"
Jul 28 18:20:18.292064 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.292002    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kube-rbac-proxy"
Jul 28 18:20:18.292064 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.292032    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="storage-initializer"
Jul 28 18:20:18.292064 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.292040    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="storage-initializer"
Jul 28 18:20:18.292247 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.292107    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kserve-container"
Jul 28 18:20:18.292247 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.292116    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba0ce12-9678-44c0-a520-89e628d76e4e" containerName="kube-rbac-proxy"
Jul 28 18:20:18.295310 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.295291    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.297749 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.297723    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:20:18.297867 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.297751    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 28 18:20:18.297867 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.297753    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:20:18.297867 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.297849    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:20:18.298840 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.298826    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:20:18.298902 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.298873    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:20:18.298942 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.298920    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:20:18.307396 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.307375    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"]
Jul 28 18:20:18.329689 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.329654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2mm\" (UniqueName: \"kubernetes.io/projected/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-api-access-xs2mm\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.329689 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.329691    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.329873 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.329760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbf72fc2-d50e-44f6-8f7b-bb438397d392-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.329873 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.329782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.329873 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.329843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.418477 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.418439    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:20:18.430432 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.430400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2mm\" (UniqueName: \"kubernetes.io/projected/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-api-access-xs2mm\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.430577 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.430438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.430577 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.430503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbf72fc2-d50e-44f6-8f7b-bb438397d392-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.430692 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.430621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.430692 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.430665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.430878 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.430857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.431273 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.431256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.431273 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.431266    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.432865 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.432844    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbf72fc2-d50e-44f6-8f7b-bb438397d392-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.438834 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.438808    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2mm\" (UniqueName: \"kubernetes.io/projected/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-api-access-xs2mm\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.605597 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.605503    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:18.739458 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:18.739375    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"]
Jul 28 18:20:18.741888 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:20:18.741862    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf72fc2_d50e_44f6_8f7b_bb438397d392.slice/crio-eeef8d8df15ec72c31e0d8b7df674aeb6a61d517d20a0d608ba388b3a674e583 WatchSource:0}: Error finding container eeef8d8df15ec72c31e0d8b7df674aeb6a61d517d20a0d608ba388b3a674e583: Status 404 returned error can't find the container with id eeef8d8df15ec72c31e0d8b7df674aeb6a61d517d20a0d608ba388b3a674e583
Jul 28 18:20:19.108722 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:19.108684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" event={"ID":"fbf72fc2-d50e-44f6-8f7b-bb438397d392","Type":"ContainerStarted","Data":"9e3360dd15ac054e4a04193e04e048e264c55fcf5bd10485e2a2e0fbe9f73954"}
Jul 28 18:20:19.109048 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:19.108742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" event={"ID":"fbf72fc2-d50e-44f6-8f7b-bb438397d392","Type":"ContainerStarted","Data":"eeef8d8df15ec72c31e0d8b7df674aeb6a61d517d20a0d608ba388b3a674e583"}
Jul 28 18:20:23.128394 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:23.128359    2583 generic.go:356] "Generic (PLEG): container finished" podID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerID="9e3360dd15ac054e4a04193e04e048e264c55fcf5bd10485e2a2e0fbe9f73954" exitCode=0
Jul 28 18:20:23.128873 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:23.128408    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" event={"ID":"fbf72fc2-d50e-44f6-8f7b-bb438397d392","Type":"ContainerDied","Data":"9e3360dd15ac054e4a04193e04e048e264c55fcf5bd10485e2a2e0fbe9f73954"}
Jul 28 18:20:24.141665 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:24.141627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" event={"ID":"fbf72fc2-d50e-44f6-8f7b-bb438397d392","Type":"ContainerStarted","Data":"10ca64396f7e96895f575759284a9a39903390821063534c97a19c26296594c7"}
Jul 28 18:20:24.142066 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:24.141717    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" event={"ID":"fbf72fc2-d50e-44f6-8f7b-bb438397d392","Type":"ContainerStarted","Data":"ba88b6c4a9ab38ff440b1580f14790ee9c62d2f2c0d4d628157fd534b8eb296d"}
Jul 28 18:20:24.142066 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:24.141905    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:24.165647 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:24.165592    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" podStartSLOduration=2.352466217 podStartE2EDuration="6.165575843s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.813109626s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:20:24.164467562 +0000 UTC m=+3790.177847457" watchObservedRunningTime="2026-07-28 18:20:24.165575843 +0000 UTC m=+3790.178955735"
Jul 28 18:20:25.145960 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:25.145932    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:20:31.156126 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:20:31.156098    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:21:01.156859 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:01.156820    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 18:21:11.156721 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:11.156680    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 18:21:21.156952 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:21.156903    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 18:21:31.157339 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:31.157304    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 18:21:41.240822 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:41.240733    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:21:48.469833 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:48.469796    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"]
Jul 28 18:21:48.470247 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:48.470044    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kserve-container" containerID="cri-o://ba88b6c4a9ab38ff440b1580f14790ee9c62d2f2c0d4d628157fd534b8eb296d" gracePeriod=30
Jul 28 18:21:48.470247 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:48.470095    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kube-rbac-proxy" containerID="cri-o://10ca64396f7e96895f575759284a9a39903390821063534c97a19c26296594c7" gracePeriod=30
Jul 28 18:21:48.525187 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:48.525160    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:49.514485 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:49.514451    2583 generic.go:356] "Generic (PLEG): container finished" podID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerID="10ca64396f7e96895f575759284a9a39903390821063534c97a19c26296594c7" exitCode=2
Jul 28 18:21:49.514870 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:49.514497    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" event={"ID":"fbf72fc2-d50e-44f6-8f7b-bb438397d392","Type":"ContainerDied","Data":"10ca64396f7e96895f575759284a9a39903390821063534c97a19c26296594c7"}
Jul 28 18:21:51.151760 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:51.151713    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 18:21:51.157225 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:51.157195    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 18:21:53.537398 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.537359    2583 generic.go:356] "Generic (PLEG): container finished" podID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerID="ba88b6c4a9ab38ff440b1580f14790ee9c62d2f2c0d4d628157fd534b8eb296d" exitCode=0
Jul 28 18:21:53.537857 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.537440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" event={"ID":"fbf72fc2-d50e-44f6-8f7b-bb438397d392","Type":"ContainerDied","Data":"ba88b6c4a9ab38ff440b1580f14790ee9c62d2f2c0d4d628157fd534b8eb296d"}
Jul 28 18:21:53.579605 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.579571    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:53.616896 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.616871    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:21:53.640503 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.640461    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs2mm\" (UniqueName: \"kubernetes.io/projected/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-api-access-xs2mm\") pod \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") "
Jul 28 18:21:53.640683 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.640509    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kserve-provision-location\") pod \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") "
Jul 28 18:21:53.640683 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.640539    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbf72fc2-d50e-44f6-8f7b-bb438397d392-proxy-tls\") pod \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") "
Jul 28 18:21:53.640683 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.640596    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-rbac-proxy-sar-config\") pod \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") "
Jul 28 18:21:53.640683 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.640673    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-cabundle-cert\") pod \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\" (UID: \"fbf72fc2-d50e-44f6-8f7b-bb438397d392\") "
Jul 28 18:21:53.640922 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.640900    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fbf72fc2-d50e-44f6-8f7b-bb438397d392" (UID: "fbf72fc2-d50e-44f6-8f7b-bb438397d392"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:21:53.641228 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.641203    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fbf72fc2-d50e-44f6-8f7b-bb438397d392" (UID: "fbf72fc2-d50e-44f6-8f7b-bb438397d392"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:21:53.641324 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.641292    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fbf72fc2-d50e-44f6-8f7b-bb438397d392" (UID: "fbf72fc2-d50e-44f6-8f7b-bb438397d392"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:21:53.643707 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.643675    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-api-access-xs2mm" (OuterVolumeSpecName: "kube-api-access-xs2mm") pod "fbf72fc2-d50e-44f6-8f7b-bb438397d392" (UID: "fbf72fc2-d50e-44f6-8f7b-bb438397d392"). InnerVolumeSpecName "kube-api-access-xs2mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:21:53.643707 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.643684    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf72fc2-d50e-44f6-8f7b-bb438397d392-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fbf72fc2-d50e-44f6-8f7b-bb438397d392" (UID: "fbf72fc2-d50e-44f6-8f7b-bb438397d392"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:21:53.737763 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.737662    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:53.741438 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.741411    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:21:53.741438 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.741435    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbf72fc2-d50e-44f6-8f7b-bb438397d392-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:21:53.741438 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.741444    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs2mm\" (UniqueName: \"kubernetes.io/projected/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kube-api-access-xs2mm\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:21:53.741733 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.741452    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbf72fc2-d50e-44f6-8f7b-bb438397d392-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:21:53.741733 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:53.741461    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbf72fc2-d50e-44f6-8f7b-bb438397d392-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:21:54.542812 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:54.542773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl" event={"ID":"fbf72fc2-d50e-44f6-8f7b-bb438397d392","Type":"ContainerDied","Data":"eeef8d8df15ec72c31e0d8b7df674aeb6a61d517d20a0d608ba388b3a674e583"}
Jul 28 18:21:54.543362 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:54.542830    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"
Jul 28 18:21:54.543362 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:54.542850    2583 scope.go:117] "RemoveContainer" containerID="10ca64396f7e96895f575759284a9a39903390821063534c97a19c26296594c7"
Jul 28 18:21:54.551795 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:54.551778    2583 scope.go:117] "RemoveContainer" containerID="ba88b6c4a9ab38ff440b1580f14790ee9c62d2f2c0d4d628157fd534b8eb296d"
Jul 28 18:21:54.558949 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:54.558933    2583 scope.go:117] "RemoveContainer" containerID="9e3360dd15ac054e4a04193e04e048e264c55fcf5bd10485e2a2e0fbe9f73954"
Jul 28 18:21:54.563116 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:54.563093    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"]
Jul 28 18:21:54.569271 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:54.569249    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl"]
Jul 28 18:21:55.659775 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:55.659739    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:56.538190 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:21:56.538155    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" path="/var/lib/kubelet/pods/fbf72fc2-d50e-44f6-8f7b-bb438397d392/volumes"
Jul 28 18:22:00.785237 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.785200    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:00.811959 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.811928    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"]
Jul 28 18:22:00.812270 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.812258    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="storage-initializer"
Jul 28 18:22:00.812317 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.812273    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="storage-initializer"
Jul 28 18:22:00.812317 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.812288    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kube-rbac-proxy"
Jul 28 18:22:00.812317 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.812293    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kube-rbac-proxy"
Jul 28 18:22:00.812317 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.812302    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kserve-container"
Jul 28 18:22:00.812317 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.812308    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kserve-container"
Jul 28 18:22:00.812489 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.812361    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kube-rbac-proxy"
Jul 28 18:22:00.812489 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.812369    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbf72fc2-d50e-44f6-8f7b-bb438397d392" containerName="kserve-container"
Jul 28 18:22:00.817216 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.817195    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:00.819619 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.819577    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 18:22:00.819959 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.819923    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:22:00.820111 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.820094    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:22:00.820187 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.820170    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 28 18:22:00.820496 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.820481    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:22:00.821039 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.820996    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:22:00.821154 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.821002    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:22:00.827178 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.827157    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"]
Jul 28 18:22:00.901659 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.901627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b363036d-ab2b-4d44-91f0-7df299034203-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:00.901659 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.901658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxwfj\" (UniqueName: \"kubernetes.io/projected/b363036d-ab2b-4d44-91f0-7df299034203-kube-api-access-pxwfj\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:00.901867 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.901682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b363036d-ab2b-4d44-91f0-7df299034203-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:00.901867 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.901704    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:00.901867 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.901799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:00.962394 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:00.962361    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:01.002231 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.002197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b363036d-ab2b-4d44-91f0-7df299034203-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:01.002399 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.002235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxwfj\" (UniqueName: \"kubernetes.io/projected/b363036d-ab2b-4d44-91f0-7df299034203-kube-api-access-pxwfj\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:01.002399 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.002259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b363036d-ab2b-4d44-91f0-7df299034203-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:01.002497 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.002479    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:01.002580 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.002555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:01.002730 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.002602    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b363036d-ab2b-4d44-91f0-7df299034203-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:01.003229 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.003206    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:01.003306 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.003208    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:01.004781 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.004760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b363036d-ab2b-4d44-91f0-7df299034203-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:01.010723 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.010691    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxwfj\" (UniqueName: \"kubernetes.io/projected/b363036d-ab2b-4d44-91f0-7df299034203-kube-api-access-pxwfj\") pod \"isvc-sklearn-predictor-5f4d89cc44-qhdm8\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:01.128652 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.128561    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:01.264792 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.264760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"]
Jul 28 18:22:01.268065 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:22:01.268038    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb363036d_ab2b_4d44_91f0_7df299034203.slice/crio-d9bd3f2a7b3f9a7e465a41b453e9cae5c93713d8a9b64840fa707655871676c9 WatchSource:0}: Error finding container d9bd3f2a7b3f9a7e465a41b453e9cae5c93713d8a9b64840fa707655871676c9: Status 404 returned error can't find the container with id d9bd3f2a7b3f9a7e465a41b453e9cae5c93713d8a9b64840fa707655871676c9
Jul 28 18:22:01.500754 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.500710    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:01.576076 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.576038    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" event={"ID":"b363036d-ab2b-4d44-91f0-7df299034203","Type":"ContainerStarted","Data":"d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a"}
Jul 28 18:22:01.576242 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:01.576101    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" event={"ID":"b363036d-ab2b-4d44-91f0-7df299034203","Type":"ContainerStarted","Data":"d9bd3f2a7b3f9a7e465a41b453e9cae5c93713d8a9b64840fa707655871676c9"}
Jul 28 18:22:05.600653 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:05.600624    2583 generic.go:356] "Generic (PLEG): container finished" podID="b363036d-ab2b-4d44-91f0-7df299034203" containerID="d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a" exitCode=0
Jul 28 18:22:05.601059 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:05.600703    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" event={"ID":"b363036d-ab2b-4d44-91f0-7df299034203","Type":"ContainerDied","Data":"d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a"}
Jul 28 18:22:06.607637 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:06.607603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" event={"ID":"b363036d-ab2b-4d44-91f0-7df299034203","Type":"ContainerStarted","Data":"eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122"}
Jul 28 18:22:06.608069 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:06.607661    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" event={"ID":"b363036d-ab2b-4d44-91f0-7df299034203","Type":"ContainerStarted","Data":"4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd"}
Jul 28 18:22:06.608069 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:06.607861    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:06.635848 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:06.635788    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podStartSLOduration=2.885390941 podStartE2EDuration="6.635768694s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.750377753s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:22:06.634288758 +0000 UTC m=+3892.647668654" watchObservedRunningTime="2026-07-28 18:22:06.635768694 +0000 UTC m=+3892.649148589"
Jul 28 18:22:07.612366 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:07.612333    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:07.613550 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:07.613521    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 18:22:08.616743 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:08.616708    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 18:22:13.621829 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:13.621799    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:22:13.622366 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:13.622342    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 18:22:23.144851 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:23.144737    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:22:23.151469 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:23.151451    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:22:23.622925 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:23.622885    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 18:22:33.622548 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:33.622509    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 18:22:43.622404 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:43.622362    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 18:22:53.622326 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:22:53.622276    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 18:23:03.622696 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:03.622658    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 18:23:13.622397 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:13.622314    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 18:23:14.846405 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:14.846376    2583 scope.go:117] "RemoveContainer" containerID="e3710338bd900ade8a06dbc0c5a3a78d842c0fa64a46993166796d33251bf993"
Jul 28 18:23:14.854377 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:14.854357    2583 scope.go:117] "RemoveContainer" containerID="ee0c27fbc58846cc0a565730aaf5cee2de1d7d5e3113940f93158e69c3575503"
Jul 28 18:23:14.861120 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:14.861100    2583 scope.go:117] "RemoveContainer" containerID="1ea80d7bb96b678afd5bbdca74626b2f2b62b6abc1e46c6428927398317f63bb"
Jul 28 18:23:18.532628 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:18.532591    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 18:23:28.540725 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:28.540698    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:23:30.996235 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:30.996203    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"]
Jul 28 18:23:30.996646 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:30.996444    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kube-rbac-proxy" containerID="cri-o://eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122" gracePeriod=30
Jul 28 18:23:30.996646 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:30.996426    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container" containerID="cri-o://4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd" gracePeriod=30
Jul 28 18:23:31.025150 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:31.025111    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:31.989554 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:31.989522    2583 generic.go:356] "Generic (PLEG): container finished" podID="b363036d-ab2b-4d44-91f0-7df299034203" containerID="eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122" exitCode=2
Jul 28 18:23:31.989737 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:31.989600    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" event={"ID":"b363036d-ab2b-4d44-91f0-7df299034203","Type":"ContainerDied","Data":"eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122"}
Jul 28 18:23:33.618043 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:33.617981    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 28 18:23:35.947446 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:35.947418    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:23:36.009195 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.009108    2583 generic.go:356] "Generic (PLEG): container finished" podID="b363036d-ab2b-4d44-91f0-7df299034203" containerID="4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd" exitCode=0
Jul 28 18:23:36.009195 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.009175    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"
Jul 28 18:23:36.009391 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.009192    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" event={"ID":"b363036d-ab2b-4d44-91f0-7df299034203","Type":"ContainerDied","Data":"4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd"}
Jul 28 18:23:36.009391 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.009257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8" event={"ID":"b363036d-ab2b-4d44-91f0-7df299034203","Type":"ContainerDied","Data":"d9bd3f2a7b3f9a7e465a41b453e9cae5c93713d8a9b64840fa707655871676c9"}
Jul 28 18:23:36.009391 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.009291    2583 scope.go:117] "RemoveContainer" containerID="eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122"
Jul 28 18:23:36.017615 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.017601    2583 scope.go:117] "RemoveContainer" containerID="4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd"
Jul 28 18:23:36.025441 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.025423    2583 scope.go:117] "RemoveContainer" containerID="d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a"
Jul 28 18:23:36.032964 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.032947    2583 scope.go:117] "RemoveContainer" containerID="eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122"
Jul 28 18:23:36.033271 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:23:36.033246    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122\": container with ID starting with eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122 not found: ID does not exist" containerID="eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122"
Jul 28 18:23:36.033342 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.033276    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122"} err="failed to get container status \"eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122\": rpc error: code = NotFound desc = could not find container \"eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122\": container with ID starting with eb1609892babd00671ee0a27db19e41d1003274d540defde4cda76f353760122 not found: ID does not exist"
Jul 28 18:23:36.033342 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.033301    2583 scope.go:117] "RemoveContainer" containerID="4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd"
Jul 28 18:23:36.033525 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:23:36.033509    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd\": container with ID starting with 4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd not found: ID does not exist" containerID="4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd"
Jul 28 18:23:36.033573 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.033530    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd"} err="failed to get container status \"4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd\": rpc error: code = NotFound desc = could not find container \"4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd\": container with ID starting with 4dc1a72e7a37ed6867b29888cb75bbb2aa8820233ddfcbb6cceb09a8245cc4cd not found: ID does not exist"
Jul 28 18:23:36.033573 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.033544    2583 scope.go:117] "RemoveContainer" containerID="d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a"
Jul 28 18:23:36.033777 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:23:36.033761    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a\": container with ID starting with d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a not found: ID does not exist" containerID="d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a"
Jul 28 18:23:36.033814 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.033785    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a"} err="failed to get container status \"d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a\": rpc error: code = NotFound desc = could not find container \"d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a\": container with ID starting with d2624c681e2d698e031ec2ac722d678dfe691fb30b23bd38cb12b92298575b3a not found: ID does not exist"
Jul 28 18:23:36.115475 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.115443    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b363036d-ab2b-4d44-91f0-7df299034203-kserve-provision-location\") pod \"b363036d-ab2b-4d44-91f0-7df299034203\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") "
Jul 28 18:23:36.115641 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.115519    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-cabundle-cert\") pod \"b363036d-ab2b-4d44-91f0-7df299034203\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") "
Jul 28 18:23:36.115641 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.115542    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxwfj\" (UniqueName: \"kubernetes.io/projected/b363036d-ab2b-4d44-91f0-7df299034203-kube-api-access-pxwfj\") pod \"b363036d-ab2b-4d44-91f0-7df299034203\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") "
Jul 28 18:23:36.115641 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.115572    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b363036d-ab2b-4d44-91f0-7df299034203-proxy-tls\") pod \"b363036d-ab2b-4d44-91f0-7df299034203\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") "
Jul 28 18:23:36.115641 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.115596    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-kube-rbac-proxy-sar-config\") pod \"b363036d-ab2b-4d44-91f0-7df299034203\" (UID: \"b363036d-ab2b-4d44-91f0-7df299034203\") "
Jul 28 18:23:36.115840 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.115793    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b363036d-ab2b-4d44-91f0-7df299034203-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b363036d-ab2b-4d44-91f0-7df299034203" (UID: "b363036d-ab2b-4d44-91f0-7df299034203"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:23:36.115942 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.115917    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b363036d-ab2b-4d44-91f0-7df299034203" (UID: "b363036d-ab2b-4d44-91f0-7df299034203"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:23:36.116077 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.116055    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b363036d-ab2b-4d44-91f0-7df299034203" (UID: "b363036d-ab2b-4d44-91f0-7df299034203"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:23:36.117820 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.117793    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b363036d-ab2b-4d44-91f0-7df299034203-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b363036d-ab2b-4d44-91f0-7df299034203" (UID: "b363036d-ab2b-4d44-91f0-7df299034203"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:23:36.117936 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.117886    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b363036d-ab2b-4d44-91f0-7df299034203-kube-api-access-pxwfj" (OuterVolumeSpecName: "kube-api-access-pxwfj") pod "b363036d-ab2b-4d44-91f0-7df299034203" (UID: "b363036d-ab2b-4d44-91f0-7df299034203"). InnerVolumeSpecName "kube-api-access-pxwfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:23:36.149038 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.148988    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:36.199127 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.199091    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"]
Jul 28 18:23:36.199442 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.199429    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kube-rbac-proxy"
Jul 28 18:23:36.199508 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.199445    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kube-rbac-proxy"
Jul 28 18:23:36.199508 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.199453    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container"
Jul 28 18:23:36.199508 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.199458    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container"
Jul 28 18:23:36.199508 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.199467    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="storage-initializer"
Jul 28 18:23:36.199508 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.199473    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="storage-initializer"
Jul 28 18:23:36.199730 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.199528    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kserve-container"
Jul 28 18:23:36.199730 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.199536    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b363036d-ab2b-4d44-91f0-7df299034203" containerName="kube-rbac-proxy"
Jul 28 18:23:36.202903 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.202882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.205739 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.205721    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 28 18:23:36.205881 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.205866    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 18:23:36.214345 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.214323    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"]
Jul 28 18:23:36.216705 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.216683    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b363036d-ab2b-4d44-91f0-7df299034203-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:23:36.216705 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.216704    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:23:36.216835 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.216715    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b363036d-ab2b-4d44-91f0-7df299034203-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:23:36.216835 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.216725    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b363036d-ab2b-4d44-91f0-7df299034203-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:23:36.216835 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.216734    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxwfj\" (UniqueName: \"kubernetes.io/projected/b363036d-ab2b-4d44-91f0-7df299034203-kube-api-access-pxwfj\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:23:36.317340 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.317309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.317482 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.317353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssx9n\" (UniqueName: \"kubernetes.io/projected/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-api-access-ssx9n\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.317482 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.317446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.317569 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.317500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.317625 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.317569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.332032 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.331989    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"]
Jul 28 18:23:36.337778 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.337748    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8"]
Jul 28 18:23:36.418353 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.418318    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.418353 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.418351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssx9n\" (UniqueName: \"kubernetes.io/projected/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-api-access-ssx9n\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.418581 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.418392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.418581 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.418411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.418581 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.418439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.418581 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:23:36.418568    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Jul 28 18:23:36.418794 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:23:36.418636    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-proxy-tls podName:de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4 nodeName:}" failed. No retries permitted until 2026-07-28 18:23:36.918615835 +0000 UTC m=+3982.931995706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" (UID: "de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Jul 28 18:23:36.418883 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.418858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.419179 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.419160    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.419226 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.419160    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.427563 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.427537    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssx9n\" (UniqueName: \"kubernetes.io/projected/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-api-access-ssx9n\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.537611 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.537580    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b363036d-ab2b-4d44-91f0-7df299034203" path="/var/lib/kubelet/pods/b363036d-ab2b-4d44-91f0-7df299034203/volumes"
Jul 28 18:23:36.553614 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.553567    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:36.923658 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.923620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:36.926249 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:36.926223    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:37.113868 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:37.113815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:37.251084 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:37.251061    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"]
Jul 28 18:23:37.253256 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:23:37.253228    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2eeda7_cb81_45f5_8eb5_1fa0bd351fc4.slice/crio-1d0abe893a76f9d4279193ee563928e04b7d1f40cb903d97105b327c41fd7277 WatchSource:0}: Error finding container 1d0abe893a76f9d4279193ee563928e04b7d1f40cb903d97105b327c41fd7277: Status 404 returned error can't find the container with id 1d0abe893a76f9d4279193ee563928e04b7d1f40cb903d97105b327c41fd7277
Jul 28 18:23:38.021117 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:38.021083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" event={"ID":"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4","Type":"ContainerStarted","Data":"115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054"}
Jul 28 18:23:38.021305 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:38.021152    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" event={"ID":"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4","Type":"ContainerStarted","Data":"1d0abe893a76f9d4279193ee563928e04b7d1f40cb903d97105b327c41fd7277"}
Jul 28 18:23:42.039550 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:42.039517    2583 generic.go:356] "Generic (PLEG): container finished" podID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerID="115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054" exitCode=0
Jul 28 18:23:42.039989 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:42.039590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" event={"ID":"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4","Type":"ContainerDied","Data":"115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054"}
Jul 28 18:23:43.046191 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:43.046144    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" event={"ID":"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4","Type":"ContainerStarted","Data":"fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08"}
Jul 28 18:23:43.046628 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:43.046216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" event={"ID":"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4","Type":"ContainerStarted","Data":"5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7"}
Jul 28 18:23:43.046628 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:43.046534    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:43.046628 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:43.046588    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:23:43.069549 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:43.069492    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" podStartSLOduration=3.125961987 podStartE2EDuration="7.069474818s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.943512831s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:23:43.068386222 +0000 UTC m=+3989.081766116" watchObservedRunningTime="2026-07-28 18:23:43.069474818 +0000 UTC m=+3989.082854710"
Jul 28 18:23:49.056662 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:23:49.056629    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:24:19.135745 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:19.135693    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 18:24:29.059657 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:29.059628    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:24:36.250964 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:36.250884    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"]
Jul 28 18:24:36.251391 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:36.251201    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="kserve-container" containerID="cri-o://5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7" gracePeriod=30
Jul 28 18:24:36.251391 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:36.251247    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="kube-rbac-proxy" containerID="cri-o://fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08" gracePeriod=30
Jul 28 18:24:36.278277 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:36.278246    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:37.283568 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:37.283532    2583 generic.go:356] "Generic (PLEG): container finished" podID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerID="fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08" exitCode=2
Jul 28 18:24:37.283960 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:37.283606    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" event={"ID":"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4","Type":"ContainerDied","Data":"fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08"}
Jul 28 18:24:39.051502 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:39.051452    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 28 18:24:41.489563 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:41.489513    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:41.618629 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:41.618592    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:43.900287 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.900259    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:24:43.983953 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.983895    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-cabundle-cert\") pod \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") "
Jul 28 18:24:43.983953 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.983953    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-proxy-tls\") pod \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") "
Jul 28 18:24:43.984206 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.984046    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssx9n\" (UniqueName: \"kubernetes.io/projected/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-api-access-ssx9n\") pod \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") "
Jul 28 18:24:43.984206 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.984174    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-rbac-proxy-sar-config\") pod \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") "
Jul 28 18:24:43.984292 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.984214    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kserve-provision-location\") pod \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\" (UID: \"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4\") "
Jul 28 18:24:43.984421 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.984383    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" (UID: "de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:24:43.984571 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.984536    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" (UID: "de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:24:43.984646 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.984568    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" (UID: "de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:24:43.984646 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.984608    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:24:43.986264 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.986245    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" (UID: "de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:24:43.986325 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:43.986255    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-api-access-ssx9n" (OuterVolumeSpecName: "kube-api-access-ssx9n") pod "de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" (UID: "de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4"). InnerVolumeSpecName "kube-api-access-ssx9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:24:44.085954 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.085915    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssx9n\" (UniqueName: \"kubernetes.io/projected/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-api-access-ssx9n\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:24:44.085954 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.085945    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:24:44.085954 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.085956    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:24:44.085954 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.085966    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:24:44.316585 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.316498    2583 generic.go:356] "Generic (PLEG): container finished" podID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerID="5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7" exitCode=0
Jul 28 18:24:44.316585 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.316560    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"
Jul 28 18:24:44.316585 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.316574    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" event={"ID":"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4","Type":"ContainerDied","Data":"5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7"}
Jul 28 18:24:44.316823 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.316633    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg" event={"ID":"de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4","Type":"ContainerDied","Data":"1d0abe893a76f9d4279193ee563928e04b7d1f40cb903d97105b327c41fd7277"}
Jul 28 18:24:44.316823 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.316659    2583 scope.go:117] "RemoveContainer" containerID="fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08"
Jul 28 18:24:44.324997 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.324980    2583 scope.go:117] "RemoveContainer" containerID="5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7"
Jul 28 18:24:44.332414 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.332398    2583 scope.go:117] "RemoveContainer" containerID="115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054"
Jul 28 18:24:44.339715 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.339696    2583 scope.go:117] "RemoveContainer" containerID="fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08"
Jul 28 18:24:44.340050 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:24:44.340009    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08\": container with ID starting with fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08 not found: ID does not exist" containerID="fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08"
Jul 28 18:24:44.340126 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.340060    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08"} err="failed to get container status \"fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08\": rpc error: code = NotFound desc = could not find container \"fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08\": container with ID starting with fe25a5300e6a73fb042a7a9228de22d04406b1239cd37e67a59eecdaa5fa9d08 not found: ID does not exist"
Jul 28 18:24:44.340126 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.340081    2583 scope.go:117] "RemoveContainer" containerID="5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7"
Jul 28 18:24:44.340282 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.340263    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"]
Jul 28 18:24:44.340342 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:24:44.340317    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7\": container with ID starting with 5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7 not found: ID does not exist" containerID="5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7"
Jul 28 18:24:44.340342 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.340332    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7"} err="failed to get container status \"5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7\": rpc error: code = NotFound desc = could not find container \"5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7\": container with ID starting with 5fa831d9c6ee7a85cdffc8c156c0b3d1a0b94412ee54ed2e7228568f451b60f7 not found: ID does not exist"
Jul 28 18:24:44.340423 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.340346    2583 scope.go:117] "RemoveContainer" containerID="115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054"
Jul 28 18:24:44.340553 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:24:44.340537    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054\": container with ID starting with 115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054 not found: ID does not exist" containerID="115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054"
Jul 28 18:24:44.340594 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.340558    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054"} err="failed to get container status \"115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054\": rpc error: code = NotFound desc = could not find container \"115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054\": container with ID starting with 115f2cc264d42190e5cb2aad9ac7f4c568c97fa08c0311ea8e40661be9441054 not found: ID does not exist"
Jul 28 18:24:44.345516 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.345496    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg"]
Jul 28 18:24:44.536997 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:24:44.536954    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" path="/var/lib/kubelet/pods/de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4/volumes"
Jul 28 18:25:01.059181 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:01.059147    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:25:01.159440 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:01.159395    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:25:14.556796 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:14.556756    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:25:19.664329 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.664286    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:25:19.714680 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.714647    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"]
Jul 28 18:25:19.714979 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.714968    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="kube-rbac-proxy"
Jul 28 18:25:19.715053 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.714981    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="kube-rbac-proxy"
Jul 28 18:25:19.715053 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.714994    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="kserve-container"
Jul 28 18:25:19.715053 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.715000    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="kserve-container"
Jul 28 18:25:19.715053 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.715012    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="storage-initializer"
Jul 28 18:25:19.715053 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.715030    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="storage-initializer"
Jul 28 18:25:19.715211 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.715079    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="kube-rbac-proxy"
Jul 28 18:25:19.715211 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.715087    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2eeda7-cb81-45f5-8eb5-1fa0bd351fc4" containerName="kserve-container"
Jul 28 18:25:19.723464 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.723442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.727012 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.726985    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:25:19.727167 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.727058    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:25:19.727167 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.727056    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 28 18:25:19.727317 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.727201    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:25:19.727986 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.727969    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:25:19.728117 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.727970    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:25:19.728117 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.727970    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:25:19.737257 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.737236    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"]
Jul 28 18:25:19.892397 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.892344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwwgb\" (UniqueName: \"kubernetes.io/projected/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-api-access-cwwgb\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.892397 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.892400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.892630 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.892455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.892630 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.892554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.892630 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.892585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.993367 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.993335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.993541 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.993372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.993541 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.993434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwwgb\" (UniqueName: \"kubernetes.io/projected/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-api-access-cwwgb\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.993541 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.993454    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.993541 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.993470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.993732 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:25:19.993582    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 28 18:25:19.993732 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:25:19.993658    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls podName:3de35732-fa7a-4e6b-8968-25f2e8a972f5 nodeName:}" failed. No retries permitted until 2026-07-28 18:25:20.493635588 +0000 UTC m=+4086.507015459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" (UID: "3de35732-fa7a-4e6b-8968-25f2e8a972f5") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 28 18:25:19.993921 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.993896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.994118 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.994103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:19.994118 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:19.994113    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:20.003838 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:20.003817    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwwgb\" (UniqueName: \"kubernetes.io/projected/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-api-access-cwwgb\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:20.497174 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:20.497138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:20.497351 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:25:20.497256    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 28 18:25:20.497351 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:25:20.497319    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls podName:3de35732-fa7a-4e6b-8968-25f2e8a972f5 nodeName:}" failed. No retries permitted until 2026-07-28 18:25:21.497302965 +0000 UTC m=+4087.510682836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" (UID: "3de35732-fa7a-4e6b-8968-25f2e8a972f5") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 28 18:25:20.660208 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:20.660163    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:25:21.504323 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:21.504279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:21.506688 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:21.506665    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:21.533934 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:21.533885    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:21.672220 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:21.672007    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"]
Jul 28 18:25:21.674894 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:25:21.674855    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de35732_fa7a_4e6b_8968_25f2e8a972f5.slice/crio-07cf456e208d587adea711e40a22bb457e3ee8478a17fe64c77de70a3934e350 WatchSource:0}: Error finding container 07cf456e208d587adea711e40a22bb457e3ee8478a17fe64c77de70a3934e350: Status 404 returned error can't find the container with id 07cf456e208d587adea711e40a22bb457e3ee8478a17fe64c77de70a3934e350
Jul 28 18:25:22.474952 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:22.474915    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" event={"ID":"3de35732-fa7a-4e6b-8968-25f2e8a972f5","Type":"ContainerStarted","Data":"3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d"}
Jul 28 18:25:22.475239 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:22.474988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" event={"ID":"3de35732-fa7a-4e6b-8968-25f2e8a972f5","Type":"ContainerStarted","Data":"07cf456e208d587adea711e40a22bb457e3ee8478a17fe64c77de70a3934e350"}
Jul 28 18:25:26.494316 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:26.494280    2583 generic.go:356] "Generic (PLEG): container finished" podID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerID="3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d" exitCode=0
Jul 28 18:25:26.494316 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:26.494324    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" event={"ID":"3de35732-fa7a-4e6b-8968-25f2e8a972f5","Type":"ContainerDied","Data":"3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d"}
Jul 28 18:25:27.500984 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:27.500947    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" event={"ID":"3de35732-fa7a-4e6b-8968-25f2e8a972f5","Type":"ContainerStarted","Data":"be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f"}
Jul 28 18:25:27.501494 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:27.501036    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" event={"ID":"3de35732-fa7a-4e6b-8968-25f2e8a972f5","Type":"ContainerStarted","Data":"320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7"}
Jul 28 18:25:27.501494 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:27.501255    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:27.529485 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:27.529431    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" podStartSLOduration=4.790537538 podStartE2EDuration="8.529412466s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.738874928s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:25:27.52743279 +0000 UTC m=+4093.540812677" watchObservedRunningTime="2026-07-28 18:25:27.529412466 +0000 UTC m=+4093.542792361"
Jul 28 18:25:28.504914 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:28.504876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:25:34.514698 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:25:34.514668    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:26:04.538862 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:04.538816    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 18:26:14.517502 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:14.517423    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:26:19.831321 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:19.831285    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"]
Jul 28 18:26:19.831737 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:19.831522    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="kserve-container" containerID="cri-o://320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7" gracePeriod=30
Jul 28 18:26:19.831737 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:19.831562    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="kube-rbac-proxy" containerID="cri-o://be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f" gracePeriod=30
Jul 28 18:26:19.840519 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:19.840489    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:20.727061 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:20.727004    2583 generic.go:356] "Generic (PLEG): container finished" podID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerID="be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f" exitCode=2
Jul 28 18:26:20.727061 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:20.727056    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" event={"ID":"3de35732-fa7a-4e6b-8968-25f2e8a972f5","Type":"ContainerDied","Data":"be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f"}
Jul 28 18:26:24.509629 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:24.509579    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 28 18:26:24.515581 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:24.515552    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 18:26:24.953562 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:24.953477    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:25.127144 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:25.127109    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:27.476569 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.476545    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:26:27.571813 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.571719    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-rbac-proxy-sar-config\") pod \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") "
Jul 28 18:26:27.571813 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.571768    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwwgb\" (UniqueName: \"kubernetes.io/projected/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-api-access-cwwgb\") pod \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") "
Jul 28 18:26:27.571813 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.571798    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls\") pod \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") "
Jul 28 18:26:27.572154 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.571834    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kserve-provision-location\") pod \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") "
Jul 28 18:26:27.572154 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.571851    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-cabundle-cert\") pod \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\" (UID: \"3de35732-fa7a-4e6b-8968-25f2e8a972f5\") "
Jul 28 18:26:27.572268 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.572236    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3de35732-fa7a-4e6b-8968-25f2e8a972f5" (UID: "3de35732-fa7a-4e6b-8968-25f2e8a972f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:27.572323 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.572258    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3de35732-fa7a-4e6b-8968-25f2e8a972f5" (UID: "3de35732-fa7a-4e6b-8968-25f2e8a972f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:26:27.572323 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.572307    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3de35732-fa7a-4e6b-8968-25f2e8a972f5" (UID: "3de35732-fa7a-4e6b-8968-25f2e8a972f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:26:27.573982 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.573961    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3de35732-fa7a-4e6b-8968-25f2e8a972f5" (UID: "3de35732-fa7a-4e6b-8968-25f2e8a972f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:26:27.574090 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.574071    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-api-access-cwwgb" (OuterVolumeSpecName: "kube-api-access-cwwgb") pod "3de35732-fa7a-4e6b-8968-25f2e8a972f5" (UID: "3de35732-fa7a-4e6b-8968-25f2e8a972f5"). InnerVolumeSpecName "kube-api-access-cwwgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:26:27.672968 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.672912    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:26:27.672968 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.672961    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwwgb\" (UniqueName: \"kubernetes.io/projected/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kube-api-access-cwwgb\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:26:27.672968 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.672975    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3de35732-fa7a-4e6b-8968-25f2e8a972f5-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:26:27.672968 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.672984    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3de35732-fa7a-4e6b-8968-25f2e8a972f5-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:26:27.672968 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.672992    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3de35732-fa7a-4e6b-8968-25f2e8a972f5-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:26:27.760192 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.760156    2583 generic.go:356] "Generic (PLEG): container finished" podID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerID="320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7" exitCode=0
Jul 28 18:26:27.760355 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.760217    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"
Jul 28 18:26:27.760355 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.760243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" event={"ID":"3de35732-fa7a-4e6b-8968-25f2e8a972f5","Type":"ContainerDied","Data":"320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7"}
Jul 28 18:26:27.760355 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.760310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6" event={"ID":"3de35732-fa7a-4e6b-8968-25f2e8a972f5","Type":"ContainerDied","Data":"07cf456e208d587adea711e40a22bb457e3ee8478a17fe64c77de70a3934e350"}
Jul 28 18:26:27.760355 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.760337    2583 scope.go:117] "RemoveContainer" containerID="be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f"
Jul 28 18:26:27.769138 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.769118    2583 scope.go:117] "RemoveContainer" containerID="320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7"
Jul 28 18:26:27.776906 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.776887    2583 scope.go:117] "RemoveContainer" containerID="3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d"
Jul 28 18:26:27.784536 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.784510    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"]
Jul 28 18:26:27.785640 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.785623    2583 scope.go:117] "RemoveContainer" containerID="be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f"
Jul 28 18:26:27.785955 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:26:27.785922    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f\": container with ID starting with be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f not found: ID does not exist" containerID="be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f"
Jul 28 18:26:27.786065 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.785963    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f"} err="failed to get container status \"be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f\": rpc error: code = NotFound desc = could not find container \"be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f\": container with ID starting with be6ceab3f8be673f21d506405900527822df0667abd182d861a2f3d76bd2f16f not found: ID does not exist"
Jul 28 18:26:27.786065 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.785988    2583 scope.go:117] "RemoveContainer" containerID="320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7"
Jul 28 18:26:27.786347 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:26:27.786325    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7\": container with ID starting with 320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7 not found: ID does not exist" containerID="320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7"
Jul 28 18:26:27.786452 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.786356    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7"} err="failed to get container status \"320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7\": rpc error: code = NotFound desc = could not find container \"320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7\": container with ID starting with 320730811f343c71727f1465b976069ad970713fd507c00852518475037fc8a7 not found: ID does not exist"
Jul 28 18:26:27.786452 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.786377    2583 scope.go:117] "RemoveContainer" containerID="3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d"
Jul 28 18:26:27.786973 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:26:27.786953    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d\": container with ID starting with 3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d not found: ID does not exist" containerID="3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d"
Jul 28 18:26:27.787088 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.786977    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d"} err="failed to get container status \"3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d\": rpc error: code = NotFound desc = could not find container \"3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d\": container with ID starting with 3689e6b27b303fc2d58b837c2c8b100c304c3665b0476ed58a7ec5954b02726d not found: ID does not exist"
Jul 28 18:26:27.788193 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:27.788176    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6"]
Jul 28 18:26:28.536685 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:26:28.536655    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" path="/var/lib/kubelet/pods/3de35732-fa7a-4e6b-8968-25f2e8a972f5/volumes"
Jul 28 18:27:23.169854 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:27:23.169744    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:27:23.178404 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:27:23.178380    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:27:55.267353 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:27:55.267276    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:00.315599 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.315563    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:00.380161 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.380128    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"]
Jul 28 18:28:00.380481 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.380469    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="storage-initializer"
Jul 28 18:28:00.380527 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.380483    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="storage-initializer"
Jul 28 18:28:00.380527 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.380492    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="kserve-container"
Jul 28 18:28:00.380527 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.380498    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="kserve-container"
Jul 28 18:28:00.380527 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.380512    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="kube-rbac-proxy"
Jul 28 18:28:00.380527 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.380518    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="kube-rbac-proxy"
Jul 28 18:28:00.380681 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.380564    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="kube-rbac-proxy"
Jul 28 18:28:00.380681 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.380573    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de35732-fa7a-4e6b-8968-25f2e8a972f5" containerName="kserve-container"
Jul 28 18:28:00.383881 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.383861    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.386620 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.386597    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:28:00.386755 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.386618    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:28:00.386755 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.386624    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 28 18:28:00.386755 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.386627    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:28:00.386755 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.386625    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:28:00.387448 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.387430    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 28 18:28:00.387546 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.387435    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:28:00.396170 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.396151    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"]
Jul 28 18:28:00.404560 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.404533    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:00.467963 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.467931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.467963 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.467967    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.468274 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.467997    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.468274 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.468014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.468274 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.468174    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx962\" (UniqueName: \"kubernetes.io/projected/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-api-access-xx962\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.569007 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.568921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx962\" (UniqueName: \"kubernetes.io/projected/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-api-access-xx962\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.569007 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.568957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.569007 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.568977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.569533 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.569034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.569533 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.569062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.569533 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.569521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.569806 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.569784    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.569846 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.569784    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.571594 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.571577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.577109 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.577082    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx962\" (UniqueName: \"kubernetes.io/projected/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-api-access-xx962\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.695113 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.695074    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:00.825417 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:00.825391    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"]
Jul 28 18:28:00.827822 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:28:00.827794    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0395c5f_db0b_4cd5_861e_474eacc2e9bf.slice/crio-f4e9826d24d13b2b5f29752cd07e6679b31aac2b3b3163ea9f995ad14c552213 WatchSource:0}: Error finding container f4e9826d24d13b2b5f29752cd07e6679b31aac2b3b3163ea9f995ad14c552213: Status 404 returned error can't find the container with id f4e9826d24d13b2b5f29752cd07e6679b31aac2b3b3163ea9f995ad14c552213
Jul 28 18:28:01.147785 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:01.147695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" event={"ID":"c0395c5f-db0b-4cd5-861e-474eacc2e9bf","Type":"ContainerStarted","Data":"898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527"}
Jul 28 18:28:01.147785 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:01.147764    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" event={"ID":"c0395c5f-db0b-4cd5-861e-474eacc2e9bf","Type":"ContainerStarted","Data":"f4e9826d24d13b2b5f29752cd07e6679b31aac2b3b3163ea9f995ad14c552213"}
Jul 28 18:28:05.165281 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:05.165244    2583 generic.go:356] "Generic (PLEG): container finished" podID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerID="898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527" exitCode=0
Jul 28 18:28:05.165681 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:05.165303    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" event={"ID":"c0395c5f-db0b-4cd5-861e-474eacc2e9bf","Type":"ContainerDied","Data":"898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527"}
Jul 28 18:28:06.171821 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:06.171786    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" event={"ID":"c0395c5f-db0b-4cd5-861e-474eacc2e9bf","Type":"ContainerStarted","Data":"3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6"}
Jul 28 18:28:06.172251 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:06.171847    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" event={"ID":"c0395c5f-db0b-4cd5-861e-474eacc2e9bf","Type":"ContainerStarted","Data":"d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff"}
Jul 28 18:28:06.172251 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:06.172118    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:06.172337 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:06.172253    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:06.173476 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:06.173453    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 18:28:06.190574 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:06.190524    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podStartSLOduration=2.641777395 podStartE2EDuration="6.190508685s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.54873129s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:28:06.190464084 +0000 UTC m=+4252.203843972" watchObservedRunningTime="2026-07-28 18:28:06.190508685 +0000 UTC m=+4252.203888579"
Jul 28 18:28:07.175986 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:07.175943    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 18:28:12.180726 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:12.180699    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:28:12.181244 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:12.181216    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 18:28:22.181150 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:22.181107    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 18:28:32.181294 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:32.181248    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 18:28:42.181402 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:42.181362    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 18:28:43.056101 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:43.056064    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:43.115506 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:43.115475    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:52.181210 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:28:52.181169    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 18:29:02.182060 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:02.182002    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 18:29:12.181613 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:12.181535    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 18:29:18.553906 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:18.553875    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:22.182186 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:22.182157    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:29:30.530567 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:30.530536    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"]
Jul 28 18:29:30.531052 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:30.530816    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container" containerID="cri-o://d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff" gracePeriod=30
Jul 28 18:29:30.531052 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:30.530839    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kube-rbac-proxy" containerID="cri-o://3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6" gracePeriod=30
Jul 28 18:29:30.561526 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:30.561497    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:31.539158 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:31.539125    2583 generic.go:356] "Generic (PLEG): container finished" podID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerID="3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6" exitCode=2
Jul 28 18:29:31.539525 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:31.539185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" event={"ID":"c0395c5f-db0b-4cd5-861e-474eacc2e9bf","Type":"ContainerDied","Data":"3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6"}
Jul 28 18:29:32.176897 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:32.176852    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 28 18:29:32.181409 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:32.181380    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 18:29:35.485393 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.485366    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:29:35.556772 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.556688    2583 generic.go:356] "Generic (PLEG): container finished" podID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerID="d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff" exitCode=0
Jul 28 18:29:35.556772 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.556754    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"
Jul 28 18:29:35.556772 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.556756    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" event={"ID":"c0395c5f-db0b-4cd5-861e-474eacc2e9bf","Type":"ContainerDied","Data":"d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff"}
Jul 28 18:29:35.557001 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.556814    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb" event={"ID":"c0395c5f-db0b-4cd5-861e-474eacc2e9bf","Type":"ContainerDied","Data":"f4e9826d24d13b2b5f29752cd07e6679b31aac2b3b3163ea9f995ad14c552213"}
Jul 28 18:29:35.557001 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.556841    2583 scope.go:117] "RemoveContainer" containerID="3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6"
Jul 28 18:29:35.564834 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.564816    2583 scope.go:117] "RemoveContainer" containerID="d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff"
Jul 28 18:29:35.571041 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.571008    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-rbac-proxy-sar-config\") pod \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") "
Jul 28 18:29:35.571150 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.571068    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kserve-provision-location\") pod \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") "
Jul 28 18:29:35.571150 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.571104    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-proxy-tls\") pod \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") "
Jul 28 18:29:35.571150 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.571135    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-cabundle-cert\") pod \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") "
Jul 28 18:29:35.571336 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.571182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx962\" (UniqueName: \"kubernetes.io/projected/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-api-access-xx962\") pod \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\" (UID: \"c0395c5f-db0b-4cd5-861e-474eacc2e9bf\") "
Jul 28 18:29:35.571392 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.571346    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0395c5f-db0b-4cd5-861e-474eacc2e9bf" (UID: "c0395c5f-db0b-4cd5-861e-474eacc2e9bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:29:35.571392 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.571348    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c0395c5f-db0b-4cd5-861e-474eacc2e9bf" (UID: "c0395c5f-db0b-4cd5-861e-474eacc2e9bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:29:35.571506 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.571485    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:29:35.571619 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.571558    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:29:35.571677 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.571631    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c0395c5f-db0b-4cd5-861e-474eacc2e9bf" (UID: "c0395c5f-db0b-4cd5-861e-474eacc2e9bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:29:35.573088 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.573069    2583 scope.go:117] "RemoveContainer" containerID="898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527"
Jul 28 18:29:35.573629 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.573600    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-api-access-xx962" (OuterVolumeSpecName: "kube-api-access-xx962") pod "c0395c5f-db0b-4cd5-861e-474eacc2e9bf" (UID: "c0395c5f-db0b-4cd5-861e-474eacc2e9bf"). InnerVolumeSpecName "kube-api-access-xx962". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:29:35.573724 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.573606    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0395c5f-db0b-4cd5-861e-474eacc2e9bf" (UID: "c0395c5f-db0b-4cd5-861e-474eacc2e9bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:29:35.583772 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.583753    2583 scope.go:117] "RemoveContainer" containerID="3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6"
Jul 28 18:29:35.584004 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:29:35.583984    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6\": container with ID starting with 3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6 not found: ID does not exist" containerID="3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6"
Jul 28 18:29:35.584069 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.584013    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6"} err="failed to get container status \"3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6\": rpc error: code = NotFound desc = could not find container \"3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6\": container with ID starting with 3187f9c5dec75b069318e9641536c703a5be54eeef7278922c5951346c4b93e6 not found: ID does not exist"
Jul 28 18:29:35.584069 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.584053    2583 scope.go:117] "RemoveContainer" containerID="d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff"
Jul 28 18:29:35.584294 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:29:35.584278    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff\": container with ID starting with d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff not found: ID does not exist" containerID="d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff"
Jul 28 18:29:35.584339 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.584298    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff"} err="failed to get container status \"d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff\": rpc error: code = NotFound desc = could not find container \"d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff\": container with ID starting with d9409ef44cfb915cc6a20534c73df9168335eaf4a498845c0653714e5f3cfdff not found: ID does not exist"
Jul 28 18:29:35.584339 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.584311    2583 scope.go:117] "RemoveContainer" containerID="898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527"
Jul 28 18:29:35.584512 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:29:35.584492    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527\": container with ID starting with 898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527 not found: ID does not exist" containerID="898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527"
Jul 28 18:29:35.584551 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.584515    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527"} err="failed to get container status \"898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527\": rpc error: code = NotFound desc = could not find container \"898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527\": container with ID starting with 898d6f6fdc51b4158f2064011a53955fbbab0e7e01347d15fec384288ab71527 not found: ID does not exist"
Jul 28 18:29:35.637276 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.637243    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:35.672366 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.672330    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx962\" (UniqueName: \"kubernetes.io/projected/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-kube-api-access-xx962\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:29:35.672366 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.672359    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:29:35.672366 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.672375    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0395c5f-db0b-4cd5-861e-474eacc2e9bf-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:29:35.717441 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.717406    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"]
Jul 28 18:29:35.717785 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.717772    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container"
Jul 28 18:29:35.717829 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.717787    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container"
Jul 28 18:29:35.717829 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.717805    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="storage-initializer"
Jul 28 18:29:35.717829 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.717812    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="storage-initializer"
Jul 28 18:29:35.717829 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.717820    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kube-rbac-proxy"
Jul 28 18:29:35.717829 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.717829    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kube-rbac-proxy"
Jul 28 18:29:35.717990 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.717878    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kube-rbac-proxy"
Jul 28 18:29:35.717990 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.717889    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" containerName="kserve-container"
Jul 28 18:29:35.722389 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.722370    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.724610 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.724589    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 28 18:29:35.724848 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.724832    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 28 18:29:35.730540 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.730518    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"]
Jul 28 18:29:35.862199 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.862109    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:35.874234 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.874210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.874354 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.874243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39f8cd49-e773-450f-8672-89b9f2458f21-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.874354 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.874343    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.874424 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.874391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmkcg\" (UniqueName: \"kubernetes.io/projected/39f8cd49-e773-450f-8672-89b9f2458f21-kube-api-access-wmkcg\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.874424 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.874410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f8cd49-e773-450f-8672-89b9f2458f21-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.880211 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.880189    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"]
Jul 28 18:29:35.884263 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.884245    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb"]
Jul 28 18:29:35.975733 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.975695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.975733 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.975746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmkcg\" (UniqueName: \"kubernetes.io/projected/39f8cd49-e773-450f-8672-89b9f2458f21-kube-api-access-wmkcg\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.975986 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.975767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f8cd49-e773-450f-8672-89b9f2458f21-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.975986 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.975819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.975986 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.975848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39f8cd49-e773-450f-8672-89b9f2458f21-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.976352 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.976330    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f8cd49-e773-450f-8672-89b9f2458f21-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.976471 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.976450    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.976568 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.976546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.978259 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.978244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39f8cd49-e773-450f-8672-89b9f2458f21-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:35.985489 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:35.985464    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmkcg\" (UniqueName: \"kubernetes.io/projected/39f8cd49-e773-450f-8672-89b9f2458f21-kube-api-access-wmkcg\") pod \"isvc-tensorflow-predictor-547dfc5669-stxfr\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:36.033891 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:36.033858    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:36.175214 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:36.175143    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"]
Jul 28 18:29:36.178388 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:29:36.178360    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f8cd49_e773_450f_8672_89b9f2458f21.slice/crio-6648cce13590b456fe89a28795544a16eaad7cf106ec9c6629cf9f447f01e14f WatchSource:0}: Error finding container 6648cce13590b456fe89a28795544a16eaad7cf106ec9c6629cf9f447f01e14f: Status 404 returned error can't find the container with id 6648cce13590b456fe89a28795544a16eaad7cf106ec9c6629cf9f447f01e14f
Jul 28 18:29:36.537420 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:36.537387    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0395c5f-db0b-4cd5-861e-474eacc2e9bf" path="/var/lib/kubelet/pods/c0395c5f-db0b-4cd5-861e-474eacc2e9bf/volumes"
Jul 28 18:29:36.562303 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:36.562265    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" event={"ID":"39f8cd49-e773-450f-8672-89b9f2458f21","Type":"ContainerStarted","Data":"34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac"}
Jul 28 18:29:36.562451 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:36.562326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" event={"ID":"39f8cd49-e773-450f-8672-89b9f2458f21","Type":"ContainerStarted","Data":"6648cce13590b456fe89a28795544a16eaad7cf106ec9c6629cf9f447f01e14f"}
Jul 28 18:29:40.580978 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:40.580944    2583 generic.go:356] "Generic (PLEG): container finished" podID="39f8cd49-e773-450f-8672-89b9f2458f21" containerID="34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac" exitCode=0
Jul 28 18:29:40.581375 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:40.581014    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" event={"ID":"39f8cd49-e773-450f-8672-89b9f2458f21","Type":"ContainerDied","Data":"34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac"}
Jul 28 18:29:40.582287 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:40.582272    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:29:44.602385 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:44.602298    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" event={"ID":"39f8cd49-e773-450f-8672-89b9f2458f21","Type":"ContainerStarted","Data":"8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304"}
Jul 28 18:29:44.602385 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:44.602362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" event={"ID":"39f8cd49-e773-450f-8672-89b9f2458f21","Type":"ContainerStarted","Data":"fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9"}
Jul 28 18:29:44.602814 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:44.602626    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:44.602814 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:44.602669    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:44.604148 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:44.604127    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 18:29:44.623578 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:44.623537    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podStartSLOduration=1.638223496 podStartE2EDuration="9.623523318s" totalImagesPullingTime="3.676241633s" totalInitContainerRuntime="4.309058189s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:29:44.622149994 +0000 UTC m=+4350.635529901" watchObservedRunningTime="2026-07-28 18:29:44.623523318 +0000 UTC m=+4350.636903211"
Jul 28 18:29:45.606175 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:45.606140    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 18:29:50.610471 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:50.610439    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:29:50.611090 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:50.611062    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 18:29:57.392441 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:29:57.392405    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:00.612205 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:00.612176    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:30:17.020207 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:17.020167    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"]
Jul 28 18:30:17.020628 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:17.020438    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kserve-container" containerID="cri-o://fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9" gracePeriod=30
Jul 28 18:30:17.020628 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:17.020483    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kube-rbac-proxy" containerID="cri-o://8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304" gracePeriod=30
Jul 28 18:30:17.045953 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:17.045928    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:17.748303 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:17.748272    2583 generic.go:356] "Generic (PLEG): container finished" podID="39f8cd49-e773-450f-8672-89b9f2458f21" containerID="8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304" exitCode=2
Jul 28 18:30:17.748480 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:17.748350    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" event={"ID":"39f8cd49-e773-450f-8672-89b9f2458f21","Type":"ContainerDied","Data":"8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304"}
Jul 28 18:30:20.607192 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:20.607152    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 18:30:22.183158 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:22.183116    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:22.778721 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:22.778684    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:25.606735 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:25.606685    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 18:30:30.606367 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:30.606325    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 18:30:30.606850 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:30.606480    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:30:35.607163 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:35.607076    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 18:30:40.606439 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:40.606385    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 18:30:45.607426 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:45.607373    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 18:30:47.669507 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.669481    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:30:47.766232 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.766195    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-kube-rbac-proxy-sar-config\") pod \"39f8cd49-e773-450f-8672-89b9f2458f21\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") "
Jul 28 18:30:47.766232 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.766248    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-cabundle-cert\") pod \"39f8cd49-e773-450f-8672-89b9f2458f21\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") "
Jul 28 18:30:47.766504 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.766276    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39f8cd49-e773-450f-8672-89b9f2458f21-proxy-tls\") pod \"39f8cd49-e773-450f-8672-89b9f2458f21\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") "
Jul 28 18:30:47.766504 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.766303    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmkcg\" (UniqueName: \"kubernetes.io/projected/39f8cd49-e773-450f-8672-89b9f2458f21-kube-api-access-wmkcg\") pod \"39f8cd49-e773-450f-8672-89b9f2458f21\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") "
Jul 28 18:30:47.766504 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.766347    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f8cd49-e773-450f-8672-89b9f2458f21-kserve-provision-location\") pod \"39f8cd49-e773-450f-8672-89b9f2458f21\" (UID: \"39f8cd49-e773-450f-8672-89b9f2458f21\") "
Jul 28 18:30:47.766698 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.766625    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "39f8cd49-e773-450f-8672-89b9f2458f21" (UID: "39f8cd49-e773-450f-8672-89b9f2458f21"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:30:47.766759 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.766716    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "39f8cd49-e773-450f-8672-89b9f2458f21" (UID: "39f8cd49-e773-450f-8672-89b9f2458f21"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:30:47.768521 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.768493    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f8cd49-e773-450f-8672-89b9f2458f21-kube-api-access-wmkcg" (OuterVolumeSpecName: "kube-api-access-wmkcg") pod "39f8cd49-e773-450f-8672-89b9f2458f21" (UID: "39f8cd49-e773-450f-8672-89b9f2458f21"). InnerVolumeSpecName "kube-api-access-wmkcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:30:47.768636 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.768601    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f8cd49-e773-450f-8672-89b9f2458f21-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39f8cd49-e773-450f-8672-89b9f2458f21" (UID: "39f8cd49-e773-450f-8672-89b9f2458f21"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:30:47.777732 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.777705    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f8cd49-e773-450f-8672-89b9f2458f21-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39f8cd49-e773-450f-8672-89b9f2458f21" (UID: "39f8cd49-e773-450f-8672-89b9f2458f21"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:47.867637 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.867552    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:30:47.867637 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.867580    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39f8cd49-e773-450f-8672-89b9f2458f21-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:30:47.867637 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.867589    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmkcg\" (UniqueName: \"kubernetes.io/projected/39f8cd49-e773-450f-8672-89b9f2458f21-kube-api-access-wmkcg\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:30:47.867637 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.867599    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f8cd49-e773-450f-8672-89b9f2458f21-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:30:47.867637 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.867609    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39f8cd49-e773-450f-8672-89b9f2458f21-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:30:47.879087 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.879053    2583 generic.go:356] "Generic (PLEG): container finished" podID="39f8cd49-e773-450f-8672-89b9f2458f21" containerID="fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9" exitCode=137
Jul 28 18:30:47.879231 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.879095    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"
Jul 28 18:30:47.879231 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.879127    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" event={"ID":"39f8cd49-e773-450f-8672-89b9f2458f21","Type":"ContainerDied","Data":"fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9"}
Jul 28 18:30:47.879231 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.879190    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr" event={"ID":"39f8cd49-e773-450f-8672-89b9f2458f21","Type":"ContainerDied","Data":"6648cce13590b456fe89a28795544a16eaad7cf106ec9c6629cf9f447f01e14f"}
Jul 28 18:30:47.879231 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.879217    2583 scope.go:117] "RemoveContainer" containerID="8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304"
Jul 28 18:30:47.888289 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.888272    2583 scope.go:117] "RemoveContainer" containerID="fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9"
Jul 28 18:30:47.895927 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.895909    2583 scope.go:117] "RemoveContainer" containerID="34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac"
Jul 28 18:30:47.903887 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.903868    2583 scope.go:117] "RemoveContainer" containerID="8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304"
Jul 28 18:30:47.904128 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.904107    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"]
Jul 28 18:30:47.904190 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:30:47.904153    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304\": container with ID starting with 8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304 not found: ID does not exist" containerID="8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304"
Jul 28 18:30:47.904190 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.904179    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304"} err="failed to get container status \"8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304\": rpc error: code = NotFound desc = could not find container \"8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304\": container with ID starting with 8d282b7bee8c477b57f5c25f9369d371ebcffbe60b756708f9fa58eb85387304 not found: ID does not exist"
Jul 28 18:30:47.904287 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.904196    2583 scope.go:117] "RemoveContainer" containerID="fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9"
Jul 28 18:30:47.904392 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:30:47.904374    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9\": container with ID starting with fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9 not found: ID does not exist" containerID="fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9"
Jul 28 18:30:47.904442 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.904400    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9"} err="failed to get container status \"fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9\": rpc error: code = NotFound desc = could not find container \"fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9\": container with ID starting with fc32446d70fdf9939066f76908d21fbe3f3df2259837c889a2f4fe70b2fed0f9 not found: ID does not exist"
Jul 28 18:30:47.904442 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.904416    2583 scope.go:117] "RemoveContainer" containerID="34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac"
Jul 28 18:30:47.904638 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:30:47.904613    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac\": container with ID starting with 34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac not found: ID does not exist" containerID="34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac"
Jul 28 18:30:47.904714 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.904640    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac"} err="failed to get container status \"34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac\": rpc error: code = NotFound desc = could not find container \"34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac\": container with ID starting with 34700e7d1fa2c4998b10ceed5e643a41c71e1b997f20639fb05bd488444080ac not found: ID does not exist"
Jul 28 18:30:47.907150 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:47.907131    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr"]
Jul 28 18:30:48.537000 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:30:48.536959    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" path="/var/lib/kubelet/pods/39f8cd49-e773-450f-8672-89b9f2458f21/volumes"
Jul 28 18:31:03.399140 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:03.399107    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:08.529685 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.529649    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:08.574825 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.574788    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"]
Jul 28 18:31:08.575202 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.575186    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="storage-initializer"
Jul 28 18:31:08.575307 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.575205    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="storage-initializer"
Jul 28 18:31:08.575307 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.575218    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kube-rbac-proxy"
Jul 28 18:31:08.575307 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.575226    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kube-rbac-proxy"
Jul 28 18:31:08.575307 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.575244    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kserve-container"
Jul 28 18:31:08.575307 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.575252    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kserve-container"
Jul 28 18:31:08.575564 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.575343    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kserve-container"
Jul 28 18:31:08.575564 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.575363    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f8cd49-e773-450f-8672-89b9f2458f21" containerName="kube-rbac-proxy"
Jul 28 18:31:08.578493 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.578472    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.582787 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.582767    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:31:08.582893 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.582836    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 28 18:31:08.582893 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.582836    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 28 18:31:08.583007 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.582997    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:31:08.583101 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.583088    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:31:08.583636 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.583616    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:31:08.583869 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.583853    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:31:08.596363 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.596335    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"]
Jul 28 18:31:08.645856 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.645827    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.645997 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.645875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9ck\" (UniqueName: \"kubernetes.io/projected/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-api-access-4m9ck\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.645997 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.645925    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2a473e-0a55-45e8-a3ab-8813a1f29b85-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.645997 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.645949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.645997 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.645987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.690546 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.690514    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:08.746767 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.746738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9ck\" (UniqueName: \"kubernetes.io/projected/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-api-access-4m9ck\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.746899 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.746778    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2a473e-0a55-45e8-a3ab-8813a1f29b85-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.746899 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.746819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.746899 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.746891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.747200 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.746930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.747391 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.747347    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.747618 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.747602    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.747768 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.747747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.749378 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.749362    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2a473e-0a55-45e8-a3ab-8813a1f29b85-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.755443 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.755412    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9ck\" (UniqueName: \"kubernetes.io/projected/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-api-access-4m9ck\") pod \"isvc-triton-predictor-5d7db58676-795zp\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:08.889012 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:08.888928    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:31:09.036076 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:09.036051    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"]
Jul 28 18:31:09.038307 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:31:09.038278    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2a473e_0a55_45e8_a3ab_8813a1f29b85.slice/crio-083065772b304cf5c4b48a18b83eabf8044f91412ea6b6880b03c4af6dee4258 WatchSource:0}: Error finding container 083065772b304cf5c4b48a18b83eabf8044f91412ea6b6880b03c4af6dee4258: Status 404 returned error can't find the container with id 083065772b304cf5c4b48a18b83eabf8044f91412ea6b6880b03c4af6dee4258
Jul 28 18:31:09.980367 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:09.980325    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" event={"ID":"de2a473e-0a55-45e8-a3ab-8813a1f29b85","Type":"ContainerStarted","Data":"7b4729505b0b3debb85ac612c43cf17d913c09ed23e1568139749fd2c141e6f7"}
Jul 28 18:31:09.980770 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:09.980388    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" event={"ID":"de2a473e-0a55-45e8-a3ab-8813a1f29b85","Type":"ContainerStarted","Data":"083065772b304cf5c4b48a18b83eabf8044f91412ea6b6880b03c4af6dee4258"}
Jul 28 18:31:12.994176 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:12.994138    2583 generic.go:356] "Generic (PLEG): container finished" podID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerID="7b4729505b0b3debb85ac612c43cf17d913c09ed23e1568139749fd2c141e6f7" exitCode=0
Jul 28 18:31:12.994606 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:31:12.994214    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" event={"ID":"de2a473e-0a55-45e8-a3ab-8813a1f29b85","Type":"ContainerDied","Data":"7b4729505b0b3debb85ac612c43cf17d913c09ed23e1568139749fd2c141e6f7"}
Jul 28 18:33:06.002641 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:06.002613    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:33:06.053648 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:06.002613    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:33:07.634852 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:07.634811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" event={"ID":"de2a473e-0a55-45e8-a3ab-8813a1f29b85","Type":"ContainerStarted","Data":"3d1cf329dc2a85a5def74b342fc6e490f29b0a5beccb7d0977045b5db6bd01a6"}
Jul 28 18:33:07.635323 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:07.634890    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:33:07.635323 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:07.635050    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:33:07.635323 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:07.635105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" event={"ID":"de2a473e-0a55-45e8-a3ab-8813a1f29b85","Type":"ContainerStarted","Data":"e5bd2fbbe13bec6a89de0c6c3d2cf82b757c318c3868f8fbbc66324f7ffffdb4"}
Jul 28 18:33:07.635972 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:07.635940    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 18:33:07.661043 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:07.660964    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" podStartSLOduration=2.034829887 podStartE2EDuration="1m59.660943494s" totalImagesPullingTime="1m53.97274828s" totalInitContainerRuntime="3.653365327s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:33:07.660591964 +0000 UTC m=+4553.673971873" watchObservedRunningTime="2026-07-28 18:33:07.660943494 +0000 UTC m=+4553.674323388"
Jul 28 18:33:08.638978 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:08.638935    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 18:33:13.643412 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:13.643384    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:33:13.644286 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:13.644268    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:33:20.041764 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:20.041726    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"]
Jul 28 18:33:20.042281 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:20.041972    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="kserve-container" containerID="cri-o://e5bd2fbbe13bec6a89de0c6c3d2cf82b757c318c3868f8fbbc66324f7ffffdb4" gracePeriod=30
Jul 28 18:33:20.042281 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:20.041995    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="kube-rbac-proxy" containerID="cri-o://3d1cf329dc2a85a5def74b342fc6e490f29b0a5beccb7d0977045b5db6bd01a6" gracePeriod=30
Jul 28 18:33:20.090745 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:20.090710    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:20.695299 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:20.695265    2583 generic.go:356] "Generic (PLEG): container finished" podID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerID="3d1cf329dc2a85a5def74b342fc6e490f29b0a5beccb7d0977045b5db6bd01a6" exitCode=2
Jul 28 18:33:20.695558 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:20.695316    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" event={"ID":"de2a473e-0a55-45e8-a3ab-8813a1f29b85","Type":"ContainerDied","Data":"3d1cf329dc2a85a5def74b342fc6e490f29b0a5beccb7d0977045b5db6bd01a6"}
Jul 28 18:33:22.706265 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:22.706232    2583 generic.go:356] "Generic (PLEG): container finished" podID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerID="e5bd2fbbe13bec6a89de0c6c3d2cf82b757c318c3868f8fbbc66324f7ffffdb4" exitCode=0
Jul 28 18:33:22.706639 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:22.706300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" event={"ID":"de2a473e-0a55-45e8-a3ab-8813a1f29b85","Type":"ContainerDied","Data":"e5bd2fbbe13bec6a89de0c6c3d2cf82b757c318c3868f8fbbc66324f7ffffdb4"}
Jul 28 18:33:23.402491 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.402468    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:33:23.499028 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.498995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2a473e-0a55-45e8-a3ab-8813a1f29b85-proxy-tls\") pod \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") "
Jul 28 18:33:23.499202 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.499065    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kserve-provision-location\") pod \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") "
Jul 28 18:33:23.499202 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.499118    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-cabundle-cert\") pod \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") "
Jul 28 18:33:23.499202 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.499144    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m9ck\" (UniqueName: \"kubernetes.io/projected/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-api-access-4m9ck\") pod \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") "
Jul 28 18:33:23.499202 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.499181    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-rbac-proxy-sar-config\") pod \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\" (UID: \"de2a473e-0a55-45e8-a3ab-8813a1f29b85\") "
Jul 28 18:33:23.499549 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.499504    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de2a473e-0a55-45e8-a3ab-8813a1f29b85" (UID: "de2a473e-0a55-45e8-a3ab-8813a1f29b85"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:33:23.499662 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.499577    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de2a473e-0a55-45e8-a3ab-8813a1f29b85" (UID: "de2a473e-0a55-45e8-a3ab-8813a1f29b85"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:33:23.499710 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.499659    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de2a473e-0a55-45e8-a3ab-8813a1f29b85" (UID: "de2a473e-0a55-45e8-a3ab-8813a1f29b85"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:33:23.501315 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.501294    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-api-access-4m9ck" (OuterVolumeSpecName: "kube-api-access-4m9ck") pod "de2a473e-0a55-45e8-a3ab-8813a1f29b85" (UID: "de2a473e-0a55-45e8-a3ab-8813a1f29b85"). InnerVolumeSpecName "kube-api-access-4m9ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:33:23.501387 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.501330    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2a473e-0a55-45e8-a3ab-8813a1f29b85-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de2a473e-0a55-45e8-a3ab-8813a1f29b85" (UID: "de2a473e-0a55-45e8-a3ab-8813a1f29b85"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:33:23.600488 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.600459    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:33:23.600488 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.600484    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m9ck\" (UniqueName: \"kubernetes.io/projected/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-api-access-4m9ck\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:33:23.600488 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.600495    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:33:23.600728 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.600505    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de2a473e-0a55-45e8-a3ab-8813a1f29b85-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:33:23.600728 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.600516    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de2a473e-0a55-45e8-a3ab-8813a1f29b85-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:33:23.712588 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.712555    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp" event={"ID":"de2a473e-0a55-45e8-a3ab-8813a1f29b85","Type":"ContainerDied","Data":"083065772b304cf5c4b48a18b83eabf8044f91412ea6b6880b03c4af6dee4258"}
Jul 28 18:33:23.712588 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.712582    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"
Jul 28 18:33:23.713078 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.712624    2583 scope.go:117] "RemoveContainer" containerID="3d1cf329dc2a85a5def74b342fc6e490f29b0a5beccb7d0977045b5db6bd01a6"
Jul 28 18:33:23.722172 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.722156    2583 scope.go:117] "RemoveContainer" containerID="e5bd2fbbe13bec6a89de0c6c3d2cf82b757c318c3868f8fbbc66324f7ffffdb4"
Jul 28 18:33:23.729970 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.729948    2583 scope.go:117] "RemoveContainer" containerID="7b4729505b0b3debb85ac612c43cf17d913c09ed23e1568139749fd2c141e6f7"
Jul 28 18:33:23.738326 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.738304    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"]
Jul 28 18:33:23.745153 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:23.745124    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp"]
Jul 28 18:33:24.537896 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:24.537856    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" path="/var/lib/kubelet/pods/de2a473e-0a55-45e8-a3ab-8813a1f29b85/volumes"
Jul 28 18:33:25.151685 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.151656    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:25.241664 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.241629    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"]
Jul 28 18:33:25.241966 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.241954    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="kube-rbac-proxy"
Jul 28 18:33:25.242010 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.241968    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="kube-rbac-proxy"
Jul 28 18:33:25.242010 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.241978    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="storage-initializer"
Jul 28 18:33:25.242010 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.241983    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="storage-initializer"
Jul 28 18:33:25.242010 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.241994    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="kserve-container"
Jul 28 18:33:25.242010 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.242000    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="kserve-container"
Jul 28 18:33:25.242195 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.242071    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="kserve-container"
Jul 28 18:33:25.242195 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.242080    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2a473e-0a55-45e8-a3ab-8813a1f29b85" containerName="kube-rbac-proxy"
Jul 28 18:33:25.245382 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.245358    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.247913 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.247885    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:33:25.248064 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.247887    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:33:25.248064 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.248038    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:33:25.248185 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.248078    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:33:25.248294 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.248278    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 18:33:25.249171 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.249153    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:33:25.249311 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.249211    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 28 18:33:25.256787 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.256762    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"]
Jul 28 18:33:25.288054 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.288004    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:25.314915 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.314880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6443b52a-b0fe-4ef3-b245-c08597eb0716-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.315122 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.314918    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpgf7\" (UniqueName: \"kubernetes.io/projected/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-api-access-zpgf7\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.315122 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.314946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.315122 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.315091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6443b52a-b0fe-4ef3-b245-c08597eb0716-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.315122 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.315118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.416009 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.415910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.416009 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.415972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6443b52a-b0fe-4ef3-b245-c08597eb0716-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.416009 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.415993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpgf7\" (UniqueName: \"kubernetes.io/projected/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-api-access-zpgf7\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.416009 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.416010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.416387 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.416117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6443b52a-b0fe-4ef3-b245-c08597eb0716-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.416643 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.416621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6443b52a-b0fe-4ef3-b245-c08597eb0716-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.416816 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.416791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.416872 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.416795    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.418448 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.418432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6443b52a-b0fe-4ef3-b245-c08597eb0716-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.424964 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.424939    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpgf7\" (UniqueName: \"kubernetes.io/projected/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-api-access-zpgf7\") pod \"isvc-xgboost-predictor-7566d65bf7-wm285\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.556584 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.556540    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:25.690075 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.689909    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"]
Jul 28 18:33:25.692995 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:33:25.692965    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6443b52a_b0fe_4ef3_b245_c08597eb0716.slice/crio-4e5aaf66b71ea7887d9db497cf63c17349d0b2a240bcd095a9d702f6c12c4937 WatchSource:0}: Error finding container 4e5aaf66b71ea7887d9db497cf63c17349d0b2a240bcd095a9d702f6c12c4937: Status 404 returned error can't find the container with id 4e5aaf66b71ea7887d9db497cf63c17349d0b2a240bcd095a9d702f6c12c4937
Jul 28 18:33:25.722888 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:25.722860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" event={"ID":"6443b52a-b0fe-4ef3-b245-c08597eb0716","Type":"ContainerStarted","Data":"4e5aaf66b71ea7887d9db497cf63c17349d0b2a240bcd095a9d702f6c12c4937"}
Jul 28 18:33:26.728364 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:26.728322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" event={"ID":"6443b52a-b0fe-4ef3-b245-c08597eb0716","Type":"ContainerStarted","Data":"33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c"}
Jul 28 18:33:29.746790 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:29.746755    2583 generic.go:356] "Generic (PLEG): container finished" podID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerID="33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c" exitCode=0
Jul 28 18:33:29.747272 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:29.746792    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" event={"ID":"6443b52a-b0fe-4ef3-b245-c08597eb0716","Type":"ContainerDied","Data":"33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c"}
Jul 28 18:33:49.845577 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:49.845494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" event={"ID":"6443b52a-b0fe-4ef3-b245-c08597eb0716","Type":"ContainerStarted","Data":"54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe"}
Jul 28 18:33:50.852265 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:50.852225    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" event={"ID":"6443b52a-b0fe-4ef3-b245-c08597eb0716","Type":"ContainerStarted","Data":"b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2"}
Jul 28 18:33:50.852662 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:50.852336    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:50.873876 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:50.873825    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podStartSLOduration=2.229059846 podStartE2EDuration="25.873809286s" totalImagesPullingTime="19.845985242s" totalInitContainerRuntime="3.798764198s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:33:50.871382313 +0000 UTC m=+4596.884762206" watchObservedRunningTime="2026-07-28 18:33:50.873809286 +0000 UTC m=+4596.887189180"
Jul 28 18:33:51.856726 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:51.856692    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:51.858133 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:51.858102    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 18:33:52.861644 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:52.861604    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 18:33:57.866772 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:57.866738    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:33:57.867260 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:33:57.867236    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 18:34:07.868078 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:34:07.868035    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 18:34:17.868241 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:34:17.868197    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 18:34:27.868180 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:34:27.868138    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 18:34:37.867324 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:34:37.867283    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 18:34:47.867184 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:34:47.867144    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 18:34:57.868974 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:34:57.868937    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:35:05.351561 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:05.351524    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"]
Jul 28 18:35:05.351952 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:05.351763    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container" containerID="cri-o://54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe" gracePeriod=30
Jul 28 18:35:05.351952 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:05.351805    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kube-rbac-proxy" containerID="cri-o://b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2" gracePeriod=30
Jul 28 18:35:05.393862 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:05.393830    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:35:06.175480 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:06.175446    2583 generic.go:356] "Generic (PLEG): container finished" podID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerID="b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2" exitCode=2
Jul 28 18:35:06.175655 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:06.175522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" event={"ID":"6443b52a-b0fe-4ef3-b245-c08597eb0716","Type":"ContainerDied","Data":"b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2"}
Jul 28 18:35:07.862667 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:07.862614    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Jul 28 18:35:07.867887 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:07.867861    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 18:35:09.103444 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.103421    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:35:09.186763 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.186676    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6443b52a-b0fe-4ef3-b245-c08597eb0716-proxy-tls\") pod \"6443b52a-b0fe-4ef3-b245-c08597eb0716\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") "
Jul 28 18:35:09.186763 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.186714    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-rbac-proxy-sar-config\") pod \"6443b52a-b0fe-4ef3-b245-c08597eb0716\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") "
Jul 28 18:35:09.186975 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.186766    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpgf7\" (UniqueName: \"kubernetes.io/projected/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-api-access-zpgf7\") pod \"6443b52a-b0fe-4ef3-b245-c08597eb0716\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") "
Jul 28 18:35:09.186975 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.186799    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-cabundle-cert\") pod \"6443b52a-b0fe-4ef3-b245-c08597eb0716\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") "
Jul 28 18:35:09.186975 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.186828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6443b52a-b0fe-4ef3-b245-c08597eb0716-kserve-provision-location\") pod \"6443b52a-b0fe-4ef3-b245-c08597eb0716\" (UID: \"6443b52a-b0fe-4ef3-b245-c08597eb0716\") "
Jul 28 18:35:09.187228 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.187197    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6443b52a-b0fe-4ef3-b245-c08597eb0716" (UID: "6443b52a-b0fe-4ef3-b245-c08597eb0716"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:35:09.187228 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.187121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6443b52a-b0fe-4ef3-b245-c08597eb0716" (UID: "6443b52a-b0fe-4ef3-b245-c08597eb0716"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:35:09.187394 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.187251    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6443b52a-b0fe-4ef3-b245-c08597eb0716-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6443b52a-b0fe-4ef3-b245-c08597eb0716" (UID: "6443b52a-b0fe-4ef3-b245-c08597eb0716"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:35:09.189355 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.189334    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6443b52a-b0fe-4ef3-b245-c08597eb0716-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6443b52a-b0fe-4ef3-b245-c08597eb0716" (UID: "6443b52a-b0fe-4ef3-b245-c08597eb0716"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:35:09.189486 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.189457    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-api-access-zpgf7" (OuterVolumeSpecName: "kube-api-access-zpgf7") pod "6443b52a-b0fe-4ef3-b245-c08597eb0716" (UID: "6443b52a-b0fe-4ef3-b245-c08597eb0716"). InnerVolumeSpecName "kube-api-access-zpgf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:35:09.191285 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.191263    2583 generic.go:356] "Generic (PLEG): container finished" podID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerID="54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe" exitCode=0
Jul 28 18:35:09.191371 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.191326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" event={"ID":"6443b52a-b0fe-4ef3-b245-c08597eb0716","Type":"ContainerDied","Data":"54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe"}
Jul 28 18:35:09.191371 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.191336    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"
Jul 28 18:35:09.191447 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.191383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285" event={"ID":"6443b52a-b0fe-4ef3-b245-c08597eb0716","Type":"ContainerDied","Data":"4e5aaf66b71ea7887d9db497cf63c17349d0b2a240bcd095a9d702f6c12c4937"}
Jul 28 18:35:09.191447 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.191409    2583 scope.go:117] "RemoveContainer" containerID="b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2"
Jul 28 18:35:09.202954 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.202939    2583 scope.go:117] "RemoveContainer" containerID="54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe"
Jul 28 18:35:09.210229 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.210210    2583 scope.go:117] "RemoveContainer" containerID="33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c"
Jul 28 18:35:09.217601 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.217584    2583 scope.go:117] "RemoveContainer" containerID="b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2"
Jul 28 18:35:09.217879 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:35:09.217860    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2\": container with ID starting with b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2 not found: ID does not exist" containerID="b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2"
Jul 28 18:35:09.217938 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.217887    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2"} err="failed to get container status \"b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2\": rpc error: code = NotFound desc = could not find container \"b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2\": container with ID starting with b1293a662b5f9f592de014bc354f1721f204fc566f64fef5b8c5798d21635fc2 not found: ID does not exist"
Jul 28 18:35:09.217938 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.217906    2583 scope.go:117] "RemoveContainer" containerID="54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe"
Jul 28 18:35:09.218258 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:35:09.218241    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe\": container with ID starting with 54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe not found: ID does not exist" containerID="54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe"
Jul 28 18:35:09.218321 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.218263    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe"} err="failed to get container status \"54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe\": rpc error: code = NotFound desc = could not find container \"54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe\": container with ID starting with 54a328d2499c380de46335aafb4d77af04ca977b53999cab48c3abcdffa977fe not found: ID does not exist"
Jul 28 18:35:09.218321 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.218277    2583 scope.go:117] "RemoveContainer" containerID="33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c"
Jul 28 18:35:09.218505 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:35:09.218488    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c\": container with ID starting with 33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c not found: ID does not exist" containerID="33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c"
Jul 28 18:35:09.218544 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.218512    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c"} err="failed to get container status \"33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c\": rpc error: code = NotFound desc = could not find container \"33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c\": container with ID starting with 33c941a9ee8ce6424a7459e4f4078b763609b113646f4060d07babc68fa5454c not found: ID does not exist"
Jul 28 18:35:09.223407 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.223386    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"]
Jul 28 18:35:09.234435 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.234416    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285"]
Jul 28 18:35:09.288302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.288273    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6443b52a-b0fe-4ef3-b245-c08597eb0716-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:35:09.288302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.288298    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:35:09.288302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.288309    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpgf7\" (UniqueName: \"kubernetes.io/projected/6443b52a-b0fe-4ef3-b245-c08597eb0716-kube-api-access-zpgf7\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:35:09.288525 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.288321    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6443b52a-b0fe-4ef3-b245-c08597eb0716-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:35:09.288525 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:09.288329    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6443b52a-b0fe-4ef3-b245-c08597eb0716-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:35:10.474505 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.474474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:35:10.537970 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.537941    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" path="/var/lib/kubelet/pods/6443b52a-b0fe-4ef3-b245-c08597eb0716/volumes"
Jul 28 18:35:10.538323 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.538307    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"]
Jul 28 18:35:10.538639 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.538627    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kube-rbac-proxy"
Jul 28 18:35:10.538676 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.538640    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kube-rbac-proxy"
Jul 28 18:35:10.538676 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.538656    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="storage-initializer"
Jul 28 18:35:10.538676 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.538661    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="storage-initializer"
Jul 28 18:35:10.538676 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.538676    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container"
Jul 28 18:35:10.538797 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.538682    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container"
Jul 28 18:35:10.538797 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.538729    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kserve-container"
Jul 28 18:35:10.538797 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.538736    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6443b52a-b0fe-4ef3-b245-c08597eb0716" containerName="kube-rbac-proxy"
Jul 28 18:35:10.541890 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.541871    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.544239 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.544216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 18:35:10.544354 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.544239    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:35:10.544354 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.544257    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:35:10.544465 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.544422    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:35:10.545111 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.545098    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:35:10.545167 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.545099    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 18:35:10.545167 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.545153    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:35:10.551879 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.551856    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"]
Jul 28 18:35:10.598824 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.598790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5xpm\" (UniqueName: \"kubernetes.io/projected/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-api-access-v5xpm\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.598824 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.598827    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eac51d91-5d26-4183-8552-c13fe4f0900a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.599088 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.598852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.599088 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.598930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.599088 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.599046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eac51d91-5d26-4183-8552-c13fe4f0900a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.700477 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.700434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eac51d91-5d26-4183-8552-c13fe4f0900a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.700645 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.700502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5xpm\" (UniqueName: \"kubernetes.io/projected/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-api-access-v5xpm\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.700645 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.700534    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eac51d91-5d26-4183-8552-c13fe4f0900a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.700645 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.700565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.700821 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.700717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.700940 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.700920    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eac51d91-5d26-4183-8552-c13fe4f0900a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.701297 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.701277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.701394 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.701301    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.702707 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.702676    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:35:10.703284 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.703265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eac51d91-5d26-4183-8552-c13fe4f0900a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.710164 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.710139    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5xpm\" (UniqueName: \"kubernetes.io/projected/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-api-access-v5xpm\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.854303 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.854197    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:10.986668 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:10.986641    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"]
Jul 28 18:35:10.988593 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:35:10.988565    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac51d91_5d26_4183_8552_c13fe4f0900a.slice/crio-4f50ec61b7385da017760791cd499a07fa123515b10f2ce042f242344d5da598 WatchSource:0}: Error finding container 4f50ec61b7385da017760791cd499a07fa123515b10f2ce042f242344d5da598: Status 404 returned error can't find the container with id 4f50ec61b7385da017760791cd499a07fa123515b10f2ce042f242344d5da598
Jul 28 18:35:11.203002 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:11.202915    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" event={"ID":"eac51d91-5d26-4183-8552-c13fe4f0900a","Type":"ContainerStarted","Data":"61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259"}
Jul 28 18:35:11.203002 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:11.202973    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" event={"ID":"eac51d91-5d26-4183-8552-c13fe4f0900a","Type":"ContainerStarted","Data":"4f50ec61b7385da017760791cd499a07fa123515b10f2ce042f242344d5da598"}
Jul 28 18:35:15.220001 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:15.219915    2583 generic.go:356] "Generic (PLEG): container finished" podID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerID="61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259" exitCode=0
Jul 28 18:35:15.220355 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:15.219990    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" event={"ID":"eac51d91-5d26-4183-8552-c13fe4f0900a","Type":"ContainerDied","Data":"61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259"}
Jul 28 18:35:16.226186 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:16.226148    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" event={"ID":"eac51d91-5d26-4183-8552-c13fe4f0900a","Type":"ContainerStarted","Data":"f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984"}
Jul 28 18:35:16.226623 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:16.226206    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" event={"ID":"eac51d91-5d26-4183-8552-c13fe4f0900a","Type":"ContainerStarted","Data":"2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d"}
Jul 28 18:35:16.226623 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:16.226449    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:16.248738 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:16.248683    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" podStartSLOduration=2.4989835400000002 podStartE2EDuration="6.248667241s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.749683701s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:35:16.246895769 +0000 UTC m=+4682.260275699" watchObservedRunningTime="2026-07-28 18:35:16.248667241 +0000 UTC m=+4682.262047133"
Jul 28 18:35:17.230550 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:17.230511    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:23.240601 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:23.240573    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:35:53.244796 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:35:53.244763    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:36:00.588459 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:00.588425    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"]
Jul 28 18:36:00.588953 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:00.588667    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="kserve-container" containerID="cri-o://2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d" gracePeriod=30
Jul 28 18:36:00.588953 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:00.588723    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="kube-rbac-proxy" containerID="cri-o://f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984" gracePeriod=30
Jul 28 18:36:00.609717 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:00.609688    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:01.426134 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:01.426101    2583 generic.go:356] "Generic (PLEG): container finished" podID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerID="f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984" exitCode=2
Jul 28 18:36:01.426313 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:01.426148    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" event={"ID":"eac51d91-5d26-4183-8552-c13fe4f0900a","Type":"ContainerDied","Data":"f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984"}
Jul 28 18:36:03.234601 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:03.234559    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 28 18:36:03.241414 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:03.241384    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 18:36:05.738138 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.738094    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:05.767940 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.767901    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"]
Jul 28 18:36:05.770454 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.770433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.772975 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.772941    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 18:36:05.772975 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.772958    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 18:36:05.781932 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.781908    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"]
Jul 28 18:36:05.834514 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.834479    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:05.855787 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.855751    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.855985 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.855795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.855985 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.855873    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30f70834-1c4d-4812-b1c0-b130f696bf89-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.855985 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.855913    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30f70834-1c4d-4812-b1c0-b130f696bf89-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.855985 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.855954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv6pv\" (UniqueName: \"kubernetes.io/projected/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-api-access-lv6pv\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.956720 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.956686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.956720 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.956721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.956985 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.956756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30f70834-1c4d-4812-b1c0-b130f696bf89-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.956985 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.956875    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30f70834-1c4d-4812-b1c0-b130f696bf89-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.956985 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.956940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv6pv\" (UniqueName: \"kubernetes.io/projected/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-api-access-lv6pv\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.957362 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.957339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30f70834-1c4d-4812-b1c0-b130f696bf89-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.957583 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.957563    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.957583 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.957579    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.959164 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.959148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30f70834-1c4d-4812-b1c0-b130f696bf89-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:05.965132 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:05.965113    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv6pv\" (UniqueName: \"kubernetes.io/projected/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-api-access-lv6pv\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xq85r\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:06.080771 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:06.080686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:06.265958 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:06.265924    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"]
Jul 28 18:36:06.268147 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:36:06.268118    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f70834_1c4d_4812_b1c0_b130f696bf89.slice/crio-a917cb8fd7b96150f40389eb305b957b4a5c11269a2e17d5b8e45003ffdb870d WatchSource:0}: Error finding container a917cb8fd7b96150f40389eb305b957b4a5c11269a2e17d5b8e45003ffdb870d: Status 404 returned error can't find the container with id a917cb8fd7b96150f40389eb305b957b4a5c11269a2e17d5b8e45003ffdb870d
Jul 28 18:36:06.449058 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:06.448943    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" event={"ID":"30f70834-1c4d-4812-b1c0-b130f696bf89","Type":"ContainerStarted","Data":"6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5"}
Jul 28 18:36:06.449058 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:06.448998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" event={"ID":"30f70834-1c4d-4812-b1c0-b130f696bf89","Type":"ContainerStarted","Data":"a917cb8fd7b96150f40389eb305b957b4a5c11269a2e17d5b8e45003ffdb870d"}
Jul 28 18:36:07.293403 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.293351    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:36:07.369972 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.369948    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eac51d91-5d26-4183-8552-c13fe4f0900a-proxy-tls\") pod \"eac51d91-5d26-4183-8552-c13fe4f0900a\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") "
Jul 28 18:36:07.370134 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.369979    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-cabundle-cert\") pod \"eac51d91-5d26-4183-8552-c13fe4f0900a\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") "
Jul 28 18:36:07.370134 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.370004    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eac51d91-5d26-4183-8552-c13fe4f0900a-kserve-provision-location\") pod \"eac51d91-5d26-4183-8552-c13fe4f0900a\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") "
Jul 28 18:36:07.370256 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.370150    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5xpm\" (UniqueName: \"kubernetes.io/projected/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-api-access-v5xpm\") pod \"eac51d91-5d26-4183-8552-c13fe4f0900a\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") "
Jul 28 18:36:07.370256 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.370190    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-rbac-proxy-sar-config\") pod \"eac51d91-5d26-4183-8552-c13fe4f0900a\" (UID: \"eac51d91-5d26-4183-8552-c13fe4f0900a\") "
Jul 28 18:36:07.370367 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.370339    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac51d91-5d26-4183-8552-c13fe4f0900a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eac51d91-5d26-4183-8552-c13fe4f0900a" (UID: "eac51d91-5d26-4183-8552-c13fe4f0900a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:36:07.370367 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.370355    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eac51d91-5d26-4183-8552-c13fe4f0900a" (UID: "eac51d91-5d26-4183-8552-c13fe4f0900a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:36:07.370531 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.370517    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:36:07.370572 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.370534    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eac51d91-5d26-4183-8552-c13fe4f0900a-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:36:07.370613 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.370570    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eac51d91-5d26-4183-8552-c13fe4f0900a" (UID: "eac51d91-5d26-4183-8552-c13fe4f0900a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:36:07.372168 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.372147    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac51d91-5d26-4183-8552-c13fe4f0900a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eac51d91-5d26-4183-8552-c13fe4f0900a" (UID: "eac51d91-5d26-4183-8552-c13fe4f0900a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:36:07.372253 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.372237    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-api-access-v5xpm" (OuterVolumeSpecName: "kube-api-access-v5xpm") pod "eac51d91-5d26-4183-8552-c13fe4f0900a" (UID: "eac51d91-5d26-4183-8552-c13fe4f0900a"). InnerVolumeSpecName "kube-api-access-v5xpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:36:07.454703 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.454668    2583 generic.go:356] "Generic (PLEG): container finished" podID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerID="2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d" exitCode=0
Jul 28 18:36:07.454861 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.454730    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"
Jul 28 18:36:07.454861 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.454745    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" event={"ID":"eac51d91-5d26-4183-8552-c13fe4f0900a","Type":"ContainerDied","Data":"2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d"}
Jul 28 18:36:07.454861 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.454814    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5" event={"ID":"eac51d91-5d26-4183-8552-c13fe4f0900a","Type":"ContainerDied","Data":"4f50ec61b7385da017760791cd499a07fa123515b10f2ce042f242344d5da598"}
Jul 28 18:36:07.454861 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.454842    2583 scope.go:117] "RemoveContainer" containerID="f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984"
Jul 28 18:36:07.464280 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.464267    2583 scope.go:117] "RemoveContainer" containerID="2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d"
Jul 28 18:36:07.470910 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.470890    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eac51d91-5d26-4183-8552-c13fe4f0900a-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:36:07.471043 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.470914    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5xpm\" (UniqueName: \"kubernetes.io/projected/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-api-access-v5xpm\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:36:07.471043 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.470928    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eac51d91-5d26-4183-8552-c13fe4f0900a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:36:07.473994 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.473976    2583 scope.go:117] "RemoveContainer" containerID="61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259"
Jul 28 18:36:07.478476 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.478450    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"]
Jul 28 18:36:07.482497 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.482479    2583 scope.go:117] "RemoveContainer" containerID="f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984"
Jul 28 18:36:07.482765 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:36:07.482744    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984\": container with ID starting with f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984 not found: ID does not exist" containerID="f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984"
Jul 28 18:36:07.482828 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.482776    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984"} err="failed to get container status \"f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984\": rpc error: code = NotFound desc = could not find container \"f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984\": container with ID starting with f3984f5b32d60919ae668497644e594f62d1fb7dec6906c3636e5673b96e5984 not found: ID does not exist"
Jul 28 18:36:07.482828 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.482792    2583 scope.go:117] "RemoveContainer" containerID="2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d"
Jul 28 18:36:07.482912 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.482867    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5"]
Jul 28 18:36:07.483030 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:36:07.483003    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d\": container with ID starting with 2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d not found: ID does not exist" containerID="2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d"
Jul 28 18:36:07.483079 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.483039    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d"} err="failed to get container status \"2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d\": rpc error: code = NotFound desc = could not find container \"2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d\": container with ID starting with 2a4bd34b7b4d9461b139c6584d79330cec787ceeec9224a6b15fbbff5b79795d not found: ID does not exist"
Jul 28 18:36:07.483079 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.483055    2583 scope.go:117] "RemoveContainer" containerID="61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259"
Jul 28 18:36:07.483288 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:36:07.483271    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259\": container with ID starting with 61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259 not found: ID does not exist" containerID="61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259"
Jul 28 18:36:07.483339 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:07.483294    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259"} err="failed to get container status \"61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259\": rpc error: code = NotFound desc = could not find container \"61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259\": container with ID starting with 61ba9986b5259bf29d4c1e50da6619f117c5b92e4a9578559a6d62f73e73a259 not found: ID does not exist"
Jul 28 18:36:08.536948 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:08.536914    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" path="/var/lib/kubelet/pods/eac51d91-5d26-4183-8552-c13fe4f0900a/volumes"
Jul 28 18:36:10.469578 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:10.469544    2583 generic.go:356] "Generic (PLEG): container finished" podID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerID="6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5" exitCode=0
Jul 28 18:36:10.469960 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:10.469616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" event={"ID":"30f70834-1c4d-4812-b1c0-b130f696bf89","Type":"ContainerDied","Data":"6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5"}
Jul 28 18:36:11.475965 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:11.475925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" event={"ID":"30f70834-1c4d-4812-b1c0-b130f696bf89","Type":"ContainerStarted","Data":"c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3"}
Jul 28 18:36:11.476391 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:11.475993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" event={"ID":"30f70834-1c4d-4812-b1c0-b130f696bf89","Type":"ContainerStarted","Data":"9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3"}
Jul 28 18:36:11.476391 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:11.476192    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:11.500069 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:11.499999    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" podStartSLOduration=2.867563333 podStartE2EDuration="6.499981784s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.632418451s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:36:11.497771847 +0000 UTC m=+4737.511151766" watchObservedRunningTime="2026-07-28 18:36:11.499981784 +0000 UTC m=+4737.513361676"
Jul 28 18:36:12.480803 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:12.480769    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:18.490986 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:18.490957    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:48.496084 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:48.495987    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:36:55.859085 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:55.859045    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"]
Jul 28 18:36:55.859517 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:55.859311    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerName="kserve-container" containerID="cri-o://9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3" gracePeriod=30
Jul 28 18:36:55.859517 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:55.859393    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerName="kube-rbac-proxy" containerID="cri-o://c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3" gracePeriod=30
Jul 28 18:36:55.879880 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:55.879851    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:56.674784 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:56.674747    2583 generic.go:356] "Generic (PLEG): container finished" podID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerID="c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3" exitCode=2
Jul 28 18:36:56.674960 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:56.674819    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" event={"ID":"30f70834-1c4d-4812-b1c0-b130f696bf89","Type":"ContainerDied","Data":"c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3"}
Jul 28 18:36:58.486495 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:36:58.486452    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 28 18:37:01.010295 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:01.010259    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:37:01.824296 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:01.824256    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:37:02.511626 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.511601    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:37:02.656513 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.656414    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-cabundle-cert\") pod \"30f70834-1c4d-4812-b1c0-b130f696bf89\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") "
Jul 28 18:37:02.656513 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.656473    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30f70834-1c4d-4812-b1c0-b130f696bf89-kserve-provision-location\") pod \"30f70834-1c4d-4812-b1c0-b130f696bf89\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") "
Jul 28 18:37:02.656513 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.656509    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30f70834-1c4d-4812-b1c0-b130f696bf89-proxy-tls\") pod \"30f70834-1c4d-4812-b1c0-b130f696bf89\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") "
Jul 28 18:37:02.656797 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.656652    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-rbac-proxy-sar-config\") pod \"30f70834-1c4d-4812-b1c0-b130f696bf89\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") "
Jul 28 18:37:02.656797 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.656694    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv6pv\" (UniqueName: \"kubernetes.io/projected/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-api-access-lv6pv\") pod \"30f70834-1c4d-4812-b1c0-b130f696bf89\" (UID: \"30f70834-1c4d-4812-b1c0-b130f696bf89\") "
Jul 28 18:37:02.656908 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.656826    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f70834-1c4d-4812-b1c0-b130f696bf89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30f70834-1c4d-4812-b1c0-b130f696bf89" (UID: "30f70834-1c4d-4812-b1c0-b130f696bf89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:37:02.656974 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.656846    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30f70834-1c4d-4812-b1c0-b130f696bf89" (UID: "30f70834-1c4d-4812-b1c0-b130f696bf89"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:37:02.657042 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.656992    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30f70834-1c4d-4812-b1c0-b130f696bf89" (UID: "30f70834-1c4d-4812-b1c0-b130f696bf89"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:37:02.657097 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.657059    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:37:02.657097 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.657074    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30f70834-1c4d-4812-b1c0-b130f696bf89-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:37:02.658711 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.658685    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f70834-1c4d-4812-b1c0-b130f696bf89-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30f70834-1c4d-4812-b1c0-b130f696bf89" (UID: "30f70834-1c4d-4812-b1c0-b130f696bf89"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:37:02.658711 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.658695    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-api-access-lv6pv" (OuterVolumeSpecName: "kube-api-access-lv6pv") pod "30f70834-1c4d-4812-b1c0-b130f696bf89" (UID: "30f70834-1c4d-4812-b1c0-b130f696bf89"). InnerVolumeSpecName "kube-api-access-lv6pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:37:02.701620 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.701586    2583 generic.go:356] "Generic (PLEG): container finished" podID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerID="9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3" exitCode=0
Jul 28 18:37:02.701817 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.701660    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"
Jul 28 18:37:02.701817 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.701669    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" event={"ID":"30f70834-1c4d-4812-b1c0-b130f696bf89","Type":"ContainerDied","Data":"9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3"}
Jul 28 18:37:02.701817 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.701772    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r" event={"ID":"30f70834-1c4d-4812-b1c0-b130f696bf89","Type":"ContainerDied","Data":"a917cb8fd7b96150f40389eb305b957b4a5c11269a2e17d5b8e45003ffdb870d"}
Jul 28 18:37:02.701817 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.701809    2583 scope.go:117] "RemoveContainer" containerID="c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3"
Jul 28 18:37:02.710469 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.710455    2583 scope.go:117] "RemoveContainer" containerID="9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3"
Jul 28 18:37:02.717771 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.717756    2583 scope.go:117] "RemoveContainer" containerID="6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5"
Jul 28 18:37:02.723969 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.723943    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"]
Jul 28 18:37:02.726143 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.725764    2583 scope.go:117] "RemoveContainer" containerID="c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3"
Jul 28 18:37:02.726394 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:37:02.726198    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3\": container with ID starting with c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3 not found: ID does not exist" containerID="c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3"
Jul 28 18:37:02.726394 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.726231    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3"} err="failed to get container status \"c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3\": rpc error: code = NotFound desc = could not find container \"c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3\": container with ID starting with c9a531ff7cdacf7bb006ac3cf5c10a66ebaa56555d97ee2789fa6b3290babcb3 not found: ID does not exist"
Jul 28 18:37:02.726394 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.726255    2583 scope.go:117] "RemoveContainer" containerID="9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3"
Jul 28 18:37:02.726574 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:37:02.726550    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3\": container with ID starting with 9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3 not found: ID does not exist" containerID="9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3"
Jul 28 18:37:02.726617 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.726578    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3"} err="failed to get container status \"9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3\": rpc error: code = NotFound desc = could not find container \"9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3\": container with ID starting with 9eeffd623dfbb45f0f5ba6b8399ea208db3f3bd511f8c11f65cec596687160c3 not found: ID does not exist"
Jul 28 18:37:02.726617 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.726592    2583 scope.go:117] "RemoveContainer" containerID="6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5"
Jul 28 18:37:02.726830 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:37:02.726812    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5\": container with ID starting with 6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5 not found: ID does not exist" containerID="6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5"
Jul 28 18:37:02.726871 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.726835    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5"} err="failed to get container status \"6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5\": rpc error: code = NotFound desc = could not find container \"6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5\": container with ID starting with 6a900d9479837bfad8d04e41a9d2320f6657404da60569e1f2a8529b2d47ecf5 not found: ID does not exist"
Jul 28 18:37:02.727746 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.727728    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r"]
Jul 28 18:37:02.758341 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.758313    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30f70834-1c4d-4812-b1c0-b130f696bf89-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:37:02.758341 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.758339    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:37:02.758525 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:02.758350    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv6pv\" (UniqueName: \"kubernetes.io/projected/30f70834-1c4d-4812-b1c0-b130f696bf89-kube-api-access-lv6pv\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:37:04.536686 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:37:04.536650    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" path="/var/lib/kubelet/pods/30f70834-1c4d-4812-b1c0-b130f696bf89/volumes"
Jul 28 18:38:06.028684 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:06.028546    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:38:06.031744 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:06.030625    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:38:22.001668 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:22.001632    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:29.673223 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:29.673183    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:41.215179 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:41.215138    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:46.291281 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.291246    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:46.360314 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360279    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"]
Jul 28 18:38:46.360723 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360680    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerName="storage-initializer"
Jul 28 18:38:46.360810 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360726    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerName="storage-initializer"
Jul 28 18:38:46.360810 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360742    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerName="kserve-container"
Jul 28 18:38:46.360810 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360771    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerName="kserve-container"
Jul 28 18:38:46.360810 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360791    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="storage-initializer"
Jul 28 18:38:46.360810 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360799    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="storage-initializer"
Jul 28 18:38:46.360993 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360819    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="kserve-container"
Jul 28 18:38:46.360993 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360827    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="kserve-container"
Jul 28 18:38:46.360993 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360841    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerName="kube-rbac-proxy"
Jul 28 18:38:46.360993 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360848    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerName="kube-rbac-proxy"
Jul 28 18:38:46.360993 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360859    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="kube-rbac-proxy"
Jul 28 18:38:46.360993 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360867    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="kube-rbac-proxy"
Jul 28 18:38:46.360993 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360957    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="kube-rbac-proxy"
Jul 28 18:38:46.360993 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360970    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac51d91-5d26-4183-8552-c13fe4f0900a" containerName="kserve-container"
Jul 28 18:38:46.360993 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360981    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerName="kserve-container"
Jul 28 18:38:46.360993 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.360990    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="30f70834-1c4d-4812-b1c0-b130f696bf89" containerName="kube-rbac-proxy"
Jul 28 18:38:46.364117 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.364103    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.366657 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.366632    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 28 18:38:46.366793 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.366706    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:38:46.366793 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.366705    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:38:46.366793 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.366726    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:38:46.366972 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.366849    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:38:46.367100 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.367084    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:38:46.367432 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.367417    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:38:46.374912 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.374887    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"]
Jul 28 18:38:46.410831 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.410796    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.411008 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.410847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a2c4fa2-ce38-4171-8376-661fb8a10526-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.411008 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.410871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a2c4fa2-ce38-4171-8376-661fb8a10526-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.411008 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.410940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.411008 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.410987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktbpw\" (UniqueName: \"kubernetes.io/projected/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-api-access-ktbpw\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.512173 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.512135    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktbpw\" (UniqueName: \"kubernetes.io/projected/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-api-access-ktbpw\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.512367 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.512186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.512367 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.512225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a2c4fa2-ce38-4171-8376-661fb8a10526-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.512367 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:38:46.512307    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 28 18:38:46.512367 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:38:46.512361    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a2c4fa2-ce38-4171-8376-661fb8a10526-proxy-tls podName:5a2c4fa2-ce38-4171-8376-661fb8a10526 nodeName:}" failed. No retries permitted until 2026-07-28 18:38:47.0123414 +0000 UTC m=+4893.025721272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a2c4fa2-ce38-4171-8376-661fb8a10526-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" (UID: "5a2c4fa2-ce38-4171-8376-661fb8a10526") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 28 18:38:46.512604 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.512389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a2c4fa2-ce38-4171-8376-661fb8a10526-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.512604 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.512443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.512758 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.512736    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a2c4fa2-ce38-4171-8376-661fb8a10526-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.512988 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.512965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.512988 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.512985    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.522746 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.522719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktbpw\" (UniqueName: \"kubernetes.io/projected/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-api-access-ktbpw\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:46.845232 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:46.845184    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:47.016867 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:47.016830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a2c4fa2-ce38-4171-8376-661fb8a10526-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:47.019336 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:47.019311    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a2c4fa2-ce38-4171-8376-661fb8a10526-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:47.275547 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:47.275501    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:47.456038 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:47.455990    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"]
Jul 28 18:38:47.460064 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:38:47.460038    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2c4fa2_ce38_4171_8376_661fb8a10526.slice/crio-8d35265be87ddb4dcf832dfa24d42d053e3fefa8c693d8d94bab3f961c7e3b30 WatchSource:0}: Error finding container 8d35265be87ddb4dcf832dfa24d42d053e3fefa8c693d8d94bab3f961c7e3b30: Status 404 returned error can't find the container with id 8d35265be87ddb4dcf832dfa24d42d053e3fefa8c693d8d94bab3f961c7e3b30
Jul 28 18:38:48.143373 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:48.143334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" event={"ID":"5a2c4fa2-ce38-4171-8376-661fb8a10526","Type":"ContainerStarted","Data":"a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810"}
Jul 28 18:38:48.143599 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:48.143393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" event={"ID":"5a2c4fa2-ce38-4171-8376-661fb8a10526","Type":"ContainerStarted","Data":"8d35265be87ddb4dcf832dfa24d42d053e3fefa8c693d8d94bab3f961c7e3b30"}
Jul 28 18:38:52.160138 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:52.160103    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerID="a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810" exitCode=0
Jul 28 18:38:52.160537 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:52.160165    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" event={"ID":"5a2c4fa2-ce38-4171-8376-661fb8a10526","Type":"ContainerDied","Data":"a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810"}
Jul 28 18:38:53.166251 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:53.166219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" event={"ID":"5a2c4fa2-ce38-4171-8376-661fb8a10526","Type":"ContainerStarted","Data":"69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d"}
Jul 28 18:38:53.166630 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:53.166282    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" event={"ID":"5a2c4fa2-ce38-4171-8376-661fb8a10526","Type":"ContainerStarted","Data":"bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5"}
Jul 28 18:38:53.166630 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:53.166476    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:38:53.187106 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:53.187048    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" podStartSLOduration=3.298232824 podStartE2EDuration="7.187005665s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.888772841s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:38:53.186335227 +0000 UTC m=+4899.199715131" watchObservedRunningTime="2026-07-28 18:38:53.187005665 +0000 UTC m=+4899.200385561"
Jul 28 18:38:54.170414 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:38:54.170380    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:39:00.179561 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:00.179532    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:39:30.236125 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:30.236084    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 18:39:40.182194 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:40.182112    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:39:46.420836 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:46.420802    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"]
Jul 28 18:39:46.421468 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:46.421061    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="kserve-container" containerID="cri-o://bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5" gracePeriod=30
Jul 28 18:39:46.421468 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:46.421109    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="kube-rbac-proxy" containerID="cri-o://69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d" gracePeriod=30
Jul 28 18:39:46.458737 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:46.458686    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:47.402388 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:47.402355    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerID="69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d" exitCode=2
Jul 28 18:39:47.402565 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:47.402412    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" event={"ID":"5a2c4fa2-ce38-4171-8376-661fb8a10526","Type":"ContainerDied","Data":"69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d"}
Jul 28 18:39:50.175129 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:50.175083    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 28 18:39:50.180521 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:50.180491    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 18:39:51.554954 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:51.554924    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:51.896329 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:51.896252    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:54.075628 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.075597    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:39:54.203795 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.203698    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a2c4fa2-ce38-4171-8376-661fb8a10526-proxy-tls\") pod \"5a2c4fa2-ce38-4171-8376-661fb8a10526\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") "
Jul 28 18:39:54.203795 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.203743    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a2c4fa2-ce38-4171-8376-661fb8a10526-kserve-provision-location\") pod \"5a2c4fa2-ce38-4171-8376-661fb8a10526\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") "
Jul 28 18:39:54.203795 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.203761    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-rbac-proxy-sar-config\") pod \"5a2c4fa2-ce38-4171-8376-661fb8a10526\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") "
Jul 28 18:39:54.203795 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.203794    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-cabundle-cert\") pod \"5a2c4fa2-ce38-4171-8376-661fb8a10526\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") "
Jul 28 18:39:54.204176 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.203913    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktbpw\" (UniqueName: \"kubernetes.io/projected/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-api-access-ktbpw\") pod \"5a2c4fa2-ce38-4171-8376-661fb8a10526\" (UID: \"5a2c4fa2-ce38-4171-8376-661fb8a10526\") "
Jul 28 18:39:54.204176 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.204088    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2c4fa2-ce38-4171-8376-661fb8a10526-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a2c4fa2-ce38-4171-8376-661fb8a10526" (UID: "5a2c4fa2-ce38-4171-8376-661fb8a10526"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:39:54.204176 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.204149    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a2c4fa2-ce38-4171-8376-661fb8a10526" (UID: "5a2c4fa2-ce38-4171-8376-661fb8a10526"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:39:54.204176 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.204155    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a2c4fa2-ce38-4171-8376-661fb8a10526" (UID: "5a2c4fa2-ce38-4171-8376-661fb8a10526"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:39:54.204319 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.204234    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a2c4fa2-ce38-4171-8376-661fb8a10526-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:39:54.204319 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.204253    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:39:54.204319 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.204266    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a2c4fa2-ce38-4171-8376-661fb8a10526-cabundle-cert\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:39:54.205961 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.205929    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a2c4fa2-ce38-4171-8376-661fb8a10526-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a2c4fa2-ce38-4171-8376-661fb8a10526" (UID: "5a2c4fa2-ce38-4171-8376-661fb8a10526"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:39:54.205961 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.205925    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-api-access-ktbpw" (OuterVolumeSpecName: "kube-api-access-ktbpw") pod "5a2c4fa2-ce38-4171-8376-661fb8a10526" (UID: "5a2c4fa2-ce38-4171-8376-661fb8a10526"). InnerVolumeSpecName "kube-api-access-ktbpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:39:54.304973 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.304933    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a2c4fa2-ce38-4171-8376-661fb8a10526-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:39:54.304973 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.304966    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktbpw\" (UniqueName: \"kubernetes.io/projected/5a2c4fa2-ce38-4171-8376-661fb8a10526-kube-api-access-ktbpw\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:39:54.433725 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.433686    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerID="bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5" exitCode=0
Jul 28 18:39:54.433922 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.433734    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" event={"ID":"5a2c4fa2-ce38-4171-8376-661fb8a10526","Type":"ContainerDied","Data":"bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5"}
Jul 28 18:39:54.433922 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.433746    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"
Jul 28 18:39:54.433922 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.433796    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt" event={"ID":"5a2c4fa2-ce38-4171-8376-661fb8a10526","Type":"ContainerDied","Data":"8d35265be87ddb4dcf832dfa24d42d053e3fefa8c693d8d94bab3f961c7e3b30"}
Jul 28 18:39:54.433922 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.433830    2583 scope.go:117] "RemoveContainer" containerID="69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d"
Jul 28 18:39:54.442276 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.442253    2583 scope.go:117] "RemoveContainer" containerID="bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5"
Jul 28 18:39:54.449838 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.449809    2583 scope.go:117] "RemoveContainer" containerID="a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810"
Jul 28 18:39:54.456950 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.456929    2583 scope.go:117] "RemoveContainer" containerID="69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d"
Jul 28 18:39:54.457242 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:39:54.457223    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d\": container with ID starting with 69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d not found: ID does not exist" containerID="69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d"
Jul 28 18:39:54.457316 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.457251    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d"} err="failed to get container status \"69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d\": rpc error: code = NotFound desc = could not find container \"69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d\": container with ID starting with 69ee01b7e754509e44ffaf369e7f2c81d41f3825d76a4f2dc5de9a434605dc1d not found: ID does not exist"
Jul 28 18:39:54.457316 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.457269    2583 scope.go:117] "RemoveContainer" containerID="bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5"
Jul 28 18:39:54.457530 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:39:54.457512    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5\": container with ID starting with bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5 not found: ID does not exist" containerID="bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5"
Jul 28 18:39:54.457599 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.457540    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5"} err="failed to get container status \"bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5\": rpc error: code = NotFound desc = could not find container \"bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5\": container with ID starting with bf41bd6391300047de28ecc653fec4b98ab15448a908851cb5f30d6d8bb2e1f5 not found: ID does not exist"
Jul 28 18:39:54.457599 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.457566    2583 scope.go:117] "RemoveContainer" containerID="a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810"
Jul 28 18:39:54.457805 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:39:54.457782    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810\": container with ID starting with a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810 not found: ID does not exist" containerID="a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810"
Jul 28 18:39:54.457887 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.457807    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810"} err="failed to get container status \"a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810\": rpc error: code = NotFound desc = could not find container \"a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810\": container with ID starting with a63a53e226861112c3b466d77c6b3ceb99d18d87c06f8f8721382f36c2b64810 not found: ID does not exist"
Jul 28 18:39:54.463761 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.463739    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"]
Jul 28 18:39:54.472719 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.472691    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt"]
Jul 28 18:39:54.537786 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:39:54.537753    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" path="/var/lib/kubelet/pods/5a2c4fa2-ce38-4171-8376-661fb8a10526/volumes"
Jul 28 18:41:11.761796 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:41:11.761714    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:41:16.859282 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:41:16.859249    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:41:17.073197 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:41:17.073152    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:42:37.777582 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:42:37.777484    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:42:37.878302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:42:37.878261    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:42:46.309571 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:42:46.309499    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:43:06.054909 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:43:06.054790    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:43:06.059846 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:43:06.057925    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:44:07.240270 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:07.240238    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:44:08.320640 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.320596    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:44:08.366522 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.366486    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"]
Jul 28 18:44:08.366803 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.366791    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="storage-initializer"
Jul 28 18:44:08.366848 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.366805    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="storage-initializer"
Jul 28 18:44:08.366848 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.366826    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="kube-rbac-proxy"
Jul 28 18:44:08.366848 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.366831    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="kube-rbac-proxy"
Jul 28 18:44:08.366848 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.366841    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="kserve-container"
Jul 28 18:44:08.366848 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.366846    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="kserve-container"
Jul 28 18:44:08.367006 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.366898    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="kube-rbac-proxy"
Jul 28 18:44:08.367006 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.366910    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2c4fa2-ce38-4171-8376-661fb8a10526" containerName="kserve-container"
Jul 28 18:44:08.370194 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.370173    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.372764 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.372730    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:44:08.372764 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.372741    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 28 18:44:08.372955 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.372753    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:44:08.372955 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.372869    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xbkwl\""
Jul 28 18:44:08.372955 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.372877    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 28 18:44:08.373114 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.373014    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:44:08.382033 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.382001    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"]
Jul 28 18:44:08.445391 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.445350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e4ee83-6946-4447-a0d3-ce61bd79e039-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.445559 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.445407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.445559 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.445486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d429b\" (UniqueName: \"kubernetes.io/projected/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-api-access-d429b\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.445559 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.445552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e4ee83-6946-4447-a0d3-ce61bd79e039-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.545995 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.545961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e4ee83-6946-4447-a0d3-ce61bd79e039-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.545995 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.546001    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.546234 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.546030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d429b\" (UniqueName: \"kubernetes.io/projected/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-api-access-d429b\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.546234 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.546060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e4ee83-6946-4447-a0d3-ce61bd79e039-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.546234 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:44:08.546172    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 28 18:44:08.546352 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:44:08.546242    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e4ee83-6946-4447-a0d3-ce61bd79e039-proxy-tls podName:99e4ee83-6946-4447-a0d3-ce61bd79e039 nodeName:}" failed. No retries permitted until 2026-07-28 18:44:09.046222301 +0000 UTC m=+5215.059602175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/99e4ee83-6946-4447-a0d3-ce61bd79e039-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5" (UID: "99e4ee83-6946-4447-a0d3-ce61bd79e039") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 28 18:44:08.546445 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.546423    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e4ee83-6946-4447-a0d3-ce61bd79e039-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.546683 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.546667    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.555631 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.555604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d429b\" (UniqueName: \"kubernetes.io/projected/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-api-access-d429b\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:08.748577 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:08.748546    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:44:09.049894 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:09.049811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e4ee83-6946-4447-a0d3-ce61bd79e039-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:09.052197 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:09.052169    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e4ee83-6946-4447-a0d3-ce61bd79e039-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:09.280972 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:09.280937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:09.408831 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:09.408804    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"]
Jul 28 18:44:09.411249 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:44:09.411223    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e4ee83_6946_4447_a0d3_ce61bd79e039.slice/crio-69d7edfe738fac4f6a8dc9efffe4355257f58fd0c218e599f8fa9bdbea611f95 WatchSource:0}: Error finding container 69d7edfe738fac4f6a8dc9efffe4355257f58fd0c218e599f8fa9bdbea611f95: Status 404 returned error can't find the container with id 69d7edfe738fac4f6a8dc9efffe4355257f58fd0c218e599f8fa9bdbea611f95
Jul 28 18:44:09.507411 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:09.507381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5" event={"ID":"99e4ee83-6946-4447-a0d3-ce61bd79e039","Type":"ContainerStarted","Data":"7995d1b97359e80bf752f8ddc4e1aaa49dd8cb46c3eb279055ba351a1c565609"}
Jul 28 18:44:09.507547 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:09.507433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5" event={"ID":"99e4ee83-6946-4447-a0d3-ce61bd79e039","Type":"ContainerStarted","Data":"69d7edfe738fac4f6a8dc9efffe4355257f58fd0c218e599f8fa9bdbea611f95"}
Jul 28 18:44:14.532013 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:14.531984    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5_99e4ee83-6946-4447-a0d3-ce61bd79e039/storage-initializer/0.log"
Jul 28 18:44:14.532401 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:14.532033    2583 generic.go:356] "Generic (PLEG): container finished" podID="99e4ee83-6946-4447-a0d3-ce61bd79e039" containerID="7995d1b97359e80bf752f8ddc4e1aaa49dd8cb46c3eb279055ba351a1c565609" exitCode=1
Jul 28 18:44:14.536539 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:14.536509    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5" event={"ID":"99e4ee83-6946-4447-a0d3-ce61bd79e039","Type":"ContainerDied","Data":"7995d1b97359e80bf752f8ddc4e1aaa49dd8cb46c3eb279055ba351a1c565609"}
Jul 28 18:44:15.538393 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:15.538365    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5_99e4ee83-6946-4447-a0d3-ce61bd79e039/storage-initializer/0.log"
Jul 28 18:44:15.538797 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:15.538490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5" event={"ID":"99e4ee83-6946-4447-a0d3-ce61bd79e039","Type":"ContainerStarted","Data":"c89b604546a48e9d256eb159358d400ced5970e7e967f806fcef3e8972446ecc"}
Jul 28 18:44:18.331709 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:18.331670    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"]
Jul 28 18:44:18.332129 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:18.331877    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5" podUID="99e4ee83-6946-4447-a0d3-ce61bd79e039" containerName="storage-initializer" containerID="cri-o://c89b604546a48e9d256eb159358d400ced5970e7e967f806fcef3e8972446ecc" gracePeriod=30
Jul 28 18:44:18.404556 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:18.404524    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:44:19.450328 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:19.450287    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:44:19.661885 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:19.661852    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:44:20.560194 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.560168    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5_99e4ee83-6946-4447-a0d3-ce61bd79e039/storage-initializer/1.log"
Jul 28 18:44:20.560617 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.560595    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5_99e4ee83-6946-4447-a0d3-ce61bd79e039/storage-initializer/0.log"
Jul 28 18:44:20.560665 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.560641    2583 generic.go:356] "Generic (PLEG): container finished" podID="99e4ee83-6946-4447-a0d3-ce61bd79e039" containerID="c89b604546a48e9d256eb159358d400ced5970e7e967f806fcef3e8972446ecc" exitCode=1
Jul 28 18:44:20.560748 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.560723    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5" event={"ID":"99e4ee83-6946-4447-a0d3-ce61bd79e039","Type":"ContainerDied","Data":"c89b604546a48e9d256eb159358d400ced5970e7e967f806fcef3e8972446ecc"}
Jul 28 18:44:20.560821 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.560808    2583 scope.go:117] "RemoveContainer" containerID="7995d1b97359e80bf752f8ddc4e1aaa49dd8cb46c3eb279055ba351a1c565609"
Jul 28 18:44:20.580199 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.580182    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5_99e4ee83-6946-4447-a0d3-ce61bd79e039/storage-initializer/1.log"
Jul 28 18:44:20.580408 ip-10-0-132-11 kubenswrapper[2583]: E0728 18:44:20.580388    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7995d1b97359e80bf752f8ddc4e1aaa49dd8cb46c3eb279055ba351a1c565609\": container with ID starting with 7995d1b97359e80bf752f8ddc4e1aaa49dd8cb46c3eb279055ba351a1c565609 not found: ID does not exist" containerID="7995d1b97359e80bf752f8ddc4e1aaa49dd8cb46c3eb279055ba351a1c565609"
Jul 28 18:44:20.583704 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.583686    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:20.756009 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.755978    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-rbac-proxy-sar-config\") pod \"99e4ee83-6946-4447-a0d3-ce61bd79e039\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") "
Jul 28 18:44:20.756224 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.756089    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e4ee83-6946-4447-a0d3-ce61bd79e039-proxy-tls\") pod \"99e4ee83-6946-4447-a0d3-ce61bd79e039\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") "
Jul 28 18:44:20.756224 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.756121    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d429b\" (UniqueName: \"kubernetes.io/projected/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-api-access-d429b\") pod \"99e4ee83-6946-4447-a0d3-ce61bd79e039\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") "
Jul 28 18:44:20.756224 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.756146    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e4ee83-6946-4447-a0d3-ce61bd79e039-kserve-provision-location\") pod \"99e4ee83-6946-4447-a0d3-ce61bd79e039\" (UID: \"99e4ee83-6946-4447-a0d3-ce61bd79e039\") "
Jul 28 18:44:20.756511 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.756432    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99e4ee83-6946-4447-a0d3-ce61bd79e039" (UID: "99e4ee83-6946-4447-a0d3-ce61bd79e039"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:44:20.756573 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.756491    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e4ee83-6946-4447-a0d3-ce61bd79e039-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99e4ee83-6946-4447-a0d3-ce61bd79e039" (UID: "99e4ee83-6946-4447-a0d3-ce61bd79e039"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:44:20.758262 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.758238    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-api-access-d429b" (OuterVolumeSpecName: "kube-api-access-d429b") pod "99e4ee83-6946-4447-a0d3-ce61bd79e039" (UID: "99e4ee83-6946-4447-a0d3-ce61bd79e039"). InnerVolumeSpecName "kube-api-access-d429b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:44:20.758337 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.758296    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e4ee83-6946-4447-a0d3-ce61bd79e039-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99e4ee83-6946-4447-a0d3-ce61bd79e039" (UID: "99e4ee83-6946-4447-a0d3-ce61bd79e039"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:44:20.857718 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.857684    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e4ee83-6946-4447-a0d3-ce61bd79e039-proxy-tls\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:44:20.857718 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.857713    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d429b\" (UniqueName: \"kubernetes.io/projected/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-api-access-d429b\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:44:20.857718 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.857722    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e4ee83-6946-4447-a0d3-ce61bd79e039-kserve-provision-location\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:44:20.857961 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:20.857731    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e4ee83-6946-4447-a0d3-ce61bd79e039-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-11.ec2.internal\" DevicePath \"\""
Jul 28 18:44:21.566358 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:21.566326    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5_99e4ee83-6946-4447-a0d3-ce61bd79e039/storage-initializer/1.log"
Jul 28 18:44:21.566838 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:21.566426    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"
Jul 28 18:44:21.566838 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:21.566429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5" event={"ID":"99e4ee83-6946-4447-a0d3-ce61bd79e039","Type":"ContainerDied","Data":"69d7edfe738fac4f6a8dc9efffe4355257f58fd0c218e599f8fa9bdbea611f95"}
Jul 28 18:44:21.566838 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:21.566479    2583 scope.go:117] "RemoveContainer" containerID="c89b604546a48e9d256eb159358d400ced5970e7e967f806fcef3e8972446ecc"
Jul 28 18:44:21.607560 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:21.607524    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"]
Jul 28 18:44:21.612883 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:21.612851    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5"]
Jul 28 18:44:22.537520 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:22.537485    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e4ee83-6946-4447-a0d3-ce61bd79e039" path="/var/lib/kubelet/pods/99e4ee83-6946-4447-a0d3-ce61bd79e039/volumes"
Jul 28 18:44:44.849660 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:44.849617    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:44:52.345906 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:44:52.345872    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:49.522862 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:45:49.522787    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:50.568796 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:45:50.568748    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:50.717886 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:45:50.717844    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:00.656878 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:46:00.656832    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:01.678299 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:46:01.678260    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:01.945554 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:46:01.945472    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:47:21.826231 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:47:21.826188    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:47:22.913147 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:47:22.913098    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:47:23.433765 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:47:23.433736    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:47:32.837149 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:47:32.837109    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:47:35.276775 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:47:35.276746    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:47:35.380217 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:47:35.380183    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:47:52.262661 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:47:52.262629    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:47:52.317888 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:47:52.317860    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:06.081714 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:06.081607    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:48:06.085749 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:06.085547    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:48:10.962186 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:10.962140    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:11.116156 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:11.116123    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:11.184849 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:11.184804    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:13.932760 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:13.932726    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-56htm_b5c24f95-13e0-4739-80ae-b6b128e520b2/global-pull-secret-syncer/0.log"
Jul 28 18:48:14.082034 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:14.081982    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log"
Jul 28 18:48:14.192285 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:14.192198    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-11.ec2.internal_a8cc528b9c87d3a6a58c64430cacff3d/haproxy/0.log"
Jul 28 18:48:16.679778 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:16.679745    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:16.743845 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:16.743811    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vmg2_ed18bab3-8bf4-4663-8f75-1f3a6fe2daf7/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:18.663557 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:18.663523    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77dadaa7-6577-46c0-ac84-002dfb51b086/alertmanager/0.log"
Jul 28 18:48:18.695764 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:18.695736    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77dadaa7-6577-46c0-ac84-002dfb51b086/config-reloader/0.log"
Jul 28 18:48:18.723132 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:18.723108    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77dadaa7-6577-46c0-ac84-002dfb51b086/kube-rbac-proxy-web/0.log"
Jul 28 18:48:18.750438 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:18.750414    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77dadaa7-6577-46c0-ac84-002dfb51b086/kube-rbac-proxy/0.log"
Jul 28 18:48:18.780369 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:18.780333    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77dadaa7-6577-46c0-ac84-002dfb51b086/kube-rbac-proxy-metric/0.log"
Jul 28 18:48:18.816510 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:18.816490    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77dadaa7-6577-46c0-ac84-002dfb51b086/prom-label-proxy/0.log"
Jul 28 18:48:18.848544 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:18.848510    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77dadaa7-6577-46c0-ac84-002dfb51b086/init-config-reloader/0.log"
Jul 28 18:48:18.931890 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:18.931812    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-fcthp_9d96b046-bc87-4458-8895-ff39f8febb53/kube-state-metrics/0.log"
Jul 28 18:48:18.971643 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:18.971612    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-fcthp_9d96b046-bc87-4458-8895-ff39f8febb53/kube-rbac-proxy-main/0.log"
Jul 28 18:48:18.997579 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:18.997552    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-fcthp_9d96b046-bc87-4458-8895-ff39f8febb53/kube-rbac-proxy-self/0.log"
Jul 28 18:48:19.187841 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:19.187763    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jlr74_991b8a71-6107-4f4c-8abc-a3136e76f7cb/node-exporter/0.log"
Jul 28 18:48:19.220144 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:19.220122    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jlr74_991b8a71-6107-4f4c-8abc-a3136e76f7cb/kube-rbac-proxy/0.log"
Jul 28 18:48:19.247336 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:19.247315    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jlr74_991b8a71-6107-4f4c-8abc-a3136e76f7cb/init-textfile/0.log"
Jul 28 18:48:19.362706 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:19.362669    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-47z74_b17bc288-c3fc-4fee-8500-f0e61dfd3729/kube-rbac-proxy-main/0.log"
Jul 28 18:48:19.390435 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:19.390408    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-47z74_b17bc288-c3fc-4fee-8500-f0e61dfd3729/kube-rbac-proxy-self/0.log"
Jul 28 18:48:19.420997 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:19.420969    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-47z74_b17bc288-c3fc-4fee-8500-f0e61dfd3729/openshift-state-metrics/0.log"
Jul 28 18:48:19.730302 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:19.730268    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-wz4jz_888fc068-1889-45eb-b4ae-bc1920e4a9ae/prometheus-operator-admission-webhook/0.log"
Jul 28 18:48:19.764011 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:19.763986    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b74b9557b-xkxbv_d5e4d4de-8e6b-4fc4-95b5-499c557118ae/telemeter-client/0.log"
Jul 28 18:48:19.793508 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:19.793484    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b74b9557b-xkxbv_d5e4d4de-8e6b-4fc4-95b5-499c557118ae/reload/0.log"
Jul 28 18:48:19.825229 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:19.825204    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b74b9557b-xkxbv_d5e4d4de-8e6b-4fc4-95b5-499c557118ae/kube-rbac-proxy/0.log"
Jul 28 18:48:22.614182 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.614148    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"]
Jul 28 18:48:22.614557 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.614454    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e4ee83-6946-4447-a0d3-ce61bd79e039" containerName="storage-initializer"
Jul 28 18:48:22.614557 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.614465    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e4ee83-6946-4447-a0d3-ce61bd79e039" containerName="storage-initializer"
Jul 28 18:48:22.614557 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.614481    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e4ee83-6946-4447-a0d3-ce61bd79e039" containerName="storage-initializer"
Jul 28 18:48:22.614557 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.614487    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e4ee83-6946-4447-a0d3-ce61bd79e039" containerName="storage-initializer"
Jul 28 18:48:22.614557 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.614541    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e4ee83-6946-4447-a0d3-ce61bd79e039" containerName="storage-initializer"
Jul 28 18:48:22.614557 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.614550    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e4ee83-6946-4447-a0d3-ce61bd79e039" containerName="storage-initializer"
Jul 28 18:48:22.617518 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.617498    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.619799 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.619780    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2g68f\"/\"openshift-service-ca.crt\""
Jul 28 18:48:22.619936 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.619915    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2g68f\"/\"kube-root-ca.crt\""
Jul 28 18:48:22.620066 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.619998    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2g68f\"/\"default-dockercfg-mjh7z\""
Jul 28 18:48:22.627883 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.627863    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"]
Jul 28 18:48:22.670276 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.670245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dmr\" (UniqueName: \"kubernetes.io/projected/a402dd62-1e71-49b8-926a-67b018ec3a1b-kube-api-access-75dmr\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.670444 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.670300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-lib-modules\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.670444 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.670323    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-proc\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.670444 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.670363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-podres\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.670444 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.670385    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-sys\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.771632 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.771596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dmr\" (UniqueName: \"kubernetes.io/projected/a402dd62-1e71-49b8-926a-67b018ec3a1b-kube-api-access-75dmr\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.771810 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.771664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-lib-modules\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.771810 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.771693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-proc\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.771810 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.771740    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-podres\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.771924 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.771797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-sys\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.771924 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.771810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-proc\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.771924 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.771837    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-podres\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.771924 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.771848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-sys\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.771924 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.771807    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a402dd62-1e71-49b8-926a-67b018ec3a1b-lib-modules\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.779557 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.779533    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dmr\" (UniqueName: \"kubernetes.io/projected/a402dd62-1e71-49b8-926a-67b018ec3a1b-kube-api-access-75dmr\") pod \"perf-node-gather-daemonset-f9xrk\" (UID: \"a402dd62-1e71-49b8-926a-67b018ec3a1b\") " pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:22.927493 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:22.927402    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:23.052496 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:23.052285    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"]
Jul 28 18:48:23.055100 ip-10-0-132-11 kubenswrapper[2583]: W0728 18:48:23.055076    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda402dd62_1e71_49b8_926a_67b018ec3a1b.slice/crio-fd3cca19b480b0640cec8096771919604196239e1733d2bd22f0b355380ee9d6 WatchSource:0}: Error finding container fd3cca19b480b0640cec8096771919604196239e1733d2bd22f0b355380ee9d6: Status 404 returned error can't find the container with id fd3cca19b480b0640cec8096771919604196239e1733d2bd22f0b355380ee9d6
Jul 28 18:48:23.584182 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:23.584146    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk" event={"ID":"a402dd62-1e71-49b8-926a-67b018ec3a1b","Type":"ContainerStarted","Data":"fe95beaf4ad73b3d192b16d744c260fad7265788c20d6fc2d43d7cf62b350a42"}
Jul 28 18:48:23.584391 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:23.584197    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk" event={"ID":"a402dd62-1e71-49b8-926a-67b018ec3a1b","Type":"ContainerStarted","Data":"fd3cca19b480b0640cec8096771919604196239e1733d2bd22f0b355380ee9d6"}
Jul 28 18:48:23.584391 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:23.584286    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:23.603051 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:23.602984    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk" podStartSLOduration=1.602968927 podStartE2EDuration="1.602968927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:48:23.602258101 +0000 UTC m=+5469.615637995" watchObservedRunningTime="2026-07-28 18:48:23.602968927 +0000 UTC m=+5469.616348820"
Jul 28 18:48:23.821439 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:23.821412    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ngl5f_3f3fb048-2464-4ef1-8f96-626c3154af68/dns/0.log"
Jul 28 18:48:23.846369 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:23.846302    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ngl5f_3f3fb048-2464-4ef1-8f96-626c3154af68/kube-rbac-proxy/0.log"
Jul 28 18:48:23.899186 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:23.899160    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hbgp6_29111c58-63e9-4233-a4ef-0983db2a6906/dns-node-resolver/0.log"
Jul 28 18:48:24.408287 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:24.408258    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5b8ffcbb54-s67n4_bcf12d19-b0be-42d4-923d-b479a41100f3/registry/0.log"
Jul 28 18:48:24.487199 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:24.487172    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sm9zr_37cc1108-c664-4196-8d56-259edfb151d5/node-ca/0.log"
Jul 28 18:48:25.649567 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:25.649534    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lq8bf_308f491b-d382-4e27-9bf5-36b72d56d07b/serve-healthcheck-canary/0.log"
Jul 28 18:48:26.214612 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:26.214585    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-slzgv_cd0d8257-ed38-4f9f-b067-c4302cac2b24/kube-rbac-proxy/0.log"
Jul 28 18:48:26.237141 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:26.237109    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-slzgv_cd0d8257-ed38-4f9f-b067-c4302cac2b24/exporter/0.log"
Jul 28 18:48:26.259299 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:26.259249    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-slzgv_cd0d8257-ed38-4f9f-b067-c4302cac2b24/extractor/0.log"
Jul 28 18:48:28.673444 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:28.673409    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-85d8b8ddfc-ftbqg_411c1fba-ee82-4396-9522-b44fa9ccfe41/manager/0.log"
Jul 28 18:48:28.708680 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:28.708640    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-gw7kp_9791465e-d449-49bf-8f13-7fb2253d53bf/manager/0.log"
Jul 28 18:48:29.303738 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:29.303714    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-6k2nm_dbc5dc0c-ae5d-4108-96e3-6d131274aa50/seaweedfs-tls-serving/0.log"
Jul 28 18:48:29.598865 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:29.598798    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2g68f/perf-node-gather-daemonset-f9xrk"
Jul 28 18:48:35.218434 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:35.218407    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kr5s_9e2236f3-3e04-44bd-acc0-6cda49391af4/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:48:35.243404 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:35.243365    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kr5s_9e2236f3-3e04-44bd-acc0-6cda49391af4/egress-router-binary-copy/0.log"
Jul 28 18:48:35.269471 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:35.269445    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kr5s_9e2236f3-3e04-44bd-acc0-6cda49391af4/cni-plugins/0.log"
Jul 28 18:48:35.294473 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:35.294447    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kr5s_9e2236f3-3e04-44bd-acc0-6cda49391af4/bond-cni-plugin/0.log"
Jul 28 18:48:35.318415 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:35.318383    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kr5s_9e2236f3-3e04-44bd-acc0-6cda49391af4/routeoverride-cni/0.log"
Jul 28 18:48:35.341927 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:35.341892    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kr5s_9e2236f3-3e04-44bd-acc0-6cda49391af4/whereabouts-cni-bincopy/0.log"
Jul 28 18:48:35.367211 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:35.367189    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kr5s_9e2236f3-3e04-44bd-acc0-6cda49391af4/whereabouts-cni/0.log"
Jul 28 18:48:35.575337 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:35.575268    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dzwxf_68a94d99-36cb-4e50-aa5c-da7ce1b387a7/kube-multus/0.log"
Jul 28 18:48:35.731853 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:35.731828    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bsr6p_67fc1b7f-80b5-41d9-9141-9bcea29579d4/network-metrics-daemon/0.log"
Jul 28 18:48:35.758318 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:35.758296    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bsr6p_67fc1b7f-80b5-41d9-9141-9bcea29579d4/kube-rbac-proxy/0.log"
Jul 28 18:48:36.783428 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:36.783400    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-controller/0.log"
Jul 28 18:48:36.803736 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:36.803711    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/0.log"
Jul 28 18:48:36.803898 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:36.803784    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovn-acl-logging/1.log"
Jul 28 18:48:36.822833 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:36.822809    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/kube-rbac-proxy-node/0.log"
Jul 28 18:48:36.848635 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:36.848610    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 18:48:36.870273 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:36.870237    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/northd/0.log"
Jul 28 18:48:36.894276 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:36.894238    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/nbdb/0.log"
Jul 28 18:48:36.917555 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:36.917521    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/sbdb/0.log"
Jul 28 18:48:36.967771 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:36.967726    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdz9h_4222a2e9-ff5a-4ebe-ac26-e782b3a58690/ovnkube-controller/0.log"
Jul 28 18:48:38.195908 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:38.195880    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qllgl_147f1d2b-fe6a-4336-a15c-6362fc89ec8f/network-check-target-container/0.log"
Jul 28 18:48:39.215695 ip-10-0-132-11 kubenswrapper[2583]: I0728 18:48:39.215664    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-pb79v_6e9eaed1-4bed-4a13-848b-d1aac51cf1c7/iptables-alerter/0.log"