Aug 04 17:59:34.516392 ip-10-0-129-212 systemd[1]: Starting Kubernetes Kubelet...
Aug 04 17:59:35.019036 ip-10-0-129-212 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:35.019036 ip-10-0-129-212 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 04 17:59:35.019036 ip-10-0-129-212 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:35.019036 ip-10-0-129-212 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 04 17:59:35.019036 ip-10-0-129-212 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:35.019759 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.019153    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 04 17:59:35.026425 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026398    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:35.026425 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026424    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:35.026425 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026429    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026432    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026436    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026441    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026443    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026446    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026450    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026453    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026456    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026458    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026462    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026466    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026469    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026472    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026474    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026477    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026481    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026485    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:35.026578 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026489    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026493    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026497    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026500    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026505    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026508    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026511    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026514    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026517    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026519    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026522    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026524    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026527    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026529    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026532    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026534    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026538    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026541    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026544    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026547    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:35.027011 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026550    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026553    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026555    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026558    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026560    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026563    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026566    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026568    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026571    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026573    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026576    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026578    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026581    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026584    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026587    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026590    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026594    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026599    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026602    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026604    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:35.027611 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026607    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026609    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026611    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026614    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026616    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026619    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026621    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026624    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026627    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026630    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026633    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026636    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026639    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026642    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026644    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026647    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026649    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026652    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026655    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026659    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:35.028138 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026662    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026664    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026667    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026670    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026673    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026675    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026678    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026680    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026683    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026685    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026688    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026691    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026693    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026696    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026698    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026701    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026704    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026707    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.026710    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:35.028629 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027508    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027517    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027520    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027523    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027526    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027534    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027537    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027540    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027542    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027545    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027552    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027555    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027558    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027560    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027563    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027565    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027568    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027571    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027575    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:35.029108 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027580    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027582    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027585    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027587    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027590    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027593    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027595    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027598    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027600    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027603    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027605    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027608    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027610    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027613    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027617    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027620    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027622    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027625    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027629    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027632    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:35.029601 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027635    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027638    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027640    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027643    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027645    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027647    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027650    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027652    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027655    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027658    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027661    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027664    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027666    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027669    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027671    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027674    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027677    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027680    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027682    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027685    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:35.030095 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027687    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027689    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027692    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027694    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027697    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027699    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027702    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027704    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027706    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027709    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027711    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027714    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027717    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027719    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027722    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027724    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027726    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027728    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027732    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027734    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:35.030597 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027736    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027739    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027741    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027744    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027746    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027749    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027751    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027760    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027764    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027767    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027770    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027774    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027776    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027779    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027783    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027787    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027789    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027792    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027795    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:35.031096 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.027797    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028225    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028230    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028233    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028236    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028239    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028242    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028245    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028248    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028250    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028253    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028256    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028258    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028261    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028265    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028267    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028270    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028272    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028274    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:35.031570 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028277    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028281    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028284    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028286    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028289    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028292    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028294    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028297    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028299    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028302    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028305    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028307    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028310    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028313    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028316    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028318    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028321    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028324    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028326    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028329    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:35.032024 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028331    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028334    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028337    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028339    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028341    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028344    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028346    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028349    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028351    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028353    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028356    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028359    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028362    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028365    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028368    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028370    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028373    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028376    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028378    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028381    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:35.032535 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028384    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028386    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028389    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028391    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028394    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028396    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028399    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028401    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028404    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028406    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028409    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028411    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028414    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028417    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028419    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028422    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028426    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028430    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028433    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028435    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:35.033059 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028438    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028442    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028445    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028448    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028451    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028455    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028458    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028464    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028467    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028470    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028472    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028475    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028477    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028480    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028482    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028485    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028487    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028490    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028492    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:35.033542 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028494    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.028497    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028584    2581 flags.go:64] FLAG: --address="0.0.0.0"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028596    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028605    2581 flags.go:64] FLAG: --anonymous-auth="true"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028612    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028631    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028636    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028641    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028646    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028649    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028652    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028655    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028658    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028662    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028664    2581 flags.go:64] FLAG: --cgroup-root=""
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028667    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028670    2581 flags.go:64] FLAG: --client-ca-file=""
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028674    2581 flags.go:64] FLAG: --cloud-provider="external"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028677    2581 flags.go:64] FLAG: --cluster-dns="[]"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028686    2581 flags.go:64] FLAG: --cluster-domain=""
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028689    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028692    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028695    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 04 17:59:35.034001 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028698    2581 flags.go:64] FLAG: --container-log-max-files="5"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028702    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028705    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028708    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028711    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028714    2581 flags.go:64] FLAG: --contention-profiling="false"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028717    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028720    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028723    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028726    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028730    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028733    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028736    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028739    2581 flags.go:64] FLAG: --enable-load-reader="false"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028742    2581 flags.go:64] FLAG: --enable-server="true"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028751    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028758    2581 flags.go:64] FLAG: --event-burst="100"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028761    2581 flags.go:64] FLAG: --event-qps="50"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028763    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028766    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028769    2581 flags.go:64] FLAG: --eviction-hard=""
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028773    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028776    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028778    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028781    2581 flags.go:64] FLAG: --eviction-soft=""
Aug 04 17:59:35.034628 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028784    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028787    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028790    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028794    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028797    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028800    2581 flags.go:64] FLAG: --fail-swap-on="true"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028803    2581 flags.go:64] FLAG: --feature-gates=""
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028806    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028809    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028812    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028815    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028818    2581 flags.go:64] FLAG: --healthz-port="10248"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028821    2581 flags.go:64] FLAG: --help="false"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028825    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028827    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028830    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028833    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028836    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028839    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028842    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028845    2581 flags.go:64] FLAG: --image-service-endpoint=""
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028848    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028850    2581 flags.go:64] FLAG: --kube-api-burst="100"
Aug 04 17:59:35.035232 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028861    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028864    2581 flags.go:64] FLAG: --kube-api-qps="50"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028867    2581 flags.go:64] FLAG: --kube-reserved=""
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028870    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028872    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028875    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028878    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028881    2581 flags.go:64] FLAG: --lock-file=""
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028883    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028886    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028889    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028895    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028898    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028901    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028904    2581 flags.go:64] FLAG: --logging-format="text"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028907    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028910    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028912    2581 flags.go:64] FLAG: --manifest-url=""
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028915    2581 flags.go:64] FLAG: --manifest-url-header=""
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028920    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028923    2581 flags.go:64] FLAG: --max-open-files="1000000"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028931    2581 flags.go:64] FLAG: --max-pods="110"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028934    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028937    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028940    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 04 17:59:35.035858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028943    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028946    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028949    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028952    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028960    2581 flags.go:64] FLAG: --node-status-max-images="50"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028963    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028966    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028969    2581 flags.go:64] FLAG: --pod-cidr=""
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028972    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028980    2581 flags.go:64] FLAG: --pod-manifest-path=""
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028982    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028986    2581 flags.go:64] FLAG: --pods-per-core="0"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028988    2581 flags.go:64] FLAG: --port="10250"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028991    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028994    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-088d66f1a30c01c32"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.028997    2581 flags.go:64] FLAG: --qos-reserved=""
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029000    2581 flags.go:64] FLAG: --read-only-port="10255"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029003    2581 flags.go:64] FLAG: --register-node="true"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029005    2581 flags.go:64] FLAG: --register-with-taints=""
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029014    2581 flags.go:64] FLAG: --registry-burst="10"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029017    2581 flags.go:64] FLAG: --registry-qps="5"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029020    2581 flags.go:64] FLAG: --reserved-cpus=""
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029022    2581 flags.go:64] FLAG: --reserved-memory=""
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029026    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029029    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 04 17:59:35.036506 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029032    2581 flags.go:64] FLAG: --rotate-certificates="false"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029035    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029037    2581 flags.go:64] FLAG: --runonce="false"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029054    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029058    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029061    2581 flags.go:64] FLAG: --seccomp-default="false"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029064    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029067    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029070    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029073    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029076    2581 flags.go:64] FLAG: --storage-driver-password="root"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029079    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029081    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029084    2581 flags.go:64] FLAG: --storage-driver-user="root"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029088    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029090    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029097    2581 flags.go:64] FLAG: --system-cgroups=""
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029100    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029105    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029108    2581 flags.go:64] FLAG: --tls-cert-file=""
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029111    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029118    2581 flags.go:64] FLAG: --tls-min-version=""
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029121    2581 flags.go:64] FLAG: --tls-private-key-file=""
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029124    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029128    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 04 17:59:35.037147 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029131    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029135    2581 flags.go:64] FLAG: --v="2"
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029139    2581 flags.go:64] FLAG: --version="false"
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029143    2581 flags.go:64] FLAG: --vmodule=""
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029147    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.029150    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029310    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029314    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029318    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029321    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029324    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029327    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029329    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029333    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029336    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029340    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029343    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029345    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029348    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029350    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029353    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:35.037744 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029355    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029358    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029362    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029365    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029367    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029370    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029373    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029375    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029378    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029381    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029385    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029387    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029391    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029393    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029396    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029398    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029401    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029403    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029406    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:35.038323 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029408    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029410    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029413    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029416    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029418    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029421    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029423    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029426    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029428    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029431    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029433    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029436    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029439    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029441    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029444    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029446    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029449    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029452    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029454    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:35.038792 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029457    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029459    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029462    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029464    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029468    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029471    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029474    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029477    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029479    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029482    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029485    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029487    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029490    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029492    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029495    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029497    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029499    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029502    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029506    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029509    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:35.039361 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029512    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029514    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029517    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029519    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029521    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029524    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029526    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029529    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029531    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029535    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029539    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029542    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029545    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029547    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029550    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029552    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029557    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029559    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029562    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029566    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:35.039849 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029568    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029571    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029573    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029576    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029578    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.029581    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.030180    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.037505    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.037529    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.037554    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.037561    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037637    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037643    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037646    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037650    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:35.040368 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037654    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037657    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037660    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037662    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037665    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037668    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037670    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037673    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037676    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037678    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037681    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037684    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037686    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037688    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037691    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037693    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037696    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037698    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037701    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037704    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:35.040749 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037707    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037709    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037712    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037714    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037717    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037721    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037724    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037726    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037729    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037731    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037734    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037736    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037739    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037741    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037744    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037746    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037749    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037752    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037754    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:35.041261 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037758    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037762    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037766    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037768    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037771    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037773    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037775    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037778    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037781    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037783    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037786    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037788    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037790    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037793    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037797    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037800    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037803    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037806    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037809    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:35.041755 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037813    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037816    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037818    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037821    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037823    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037825    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037828    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037830    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037832    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037835    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037837    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037840    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037843    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037845    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037848    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037850    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037852    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037855    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037857    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037860    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:35.042253 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037862    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037864    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037868    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037872    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037874    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037877    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037879    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037882    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037884    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037887    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037891    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037894    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037897    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037900    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037902    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037905    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:35.042728 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.037907    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.037912    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038034    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038059    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038063    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038068    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038072    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038076    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038080    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038083    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038085    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038088    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038091    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038094    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038096    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:35.043129 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038099    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038102    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038104    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038108    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038110    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038113    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038115    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038117    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038120    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038123    2581 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038125    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038128    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038132    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038136    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038139    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038143    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038146    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038149    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038152    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:35.043497 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038155    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038157    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038160    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038162    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038165    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038167    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038170    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038172    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038174    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038177    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038181    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038184    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038187    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038190    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038193    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038196    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038198    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038201    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038213    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:35.043965 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038216    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038219    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038221    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038225    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038227    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038230    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038233    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038236    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038239    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038241    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038244    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038247    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038250    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038253    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038255    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038258    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038260    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038262    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038265    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038267    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:35.044572 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038270    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038272    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038275    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038277    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038280    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038282    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038285    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038288    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038290    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038294    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038298    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038301    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038303    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038306    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038308    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038311    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038313    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038316    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038318    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038321    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:35.045061 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038324    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038326    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038329    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038332    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038335    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038337    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038340    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:35.038342    2581 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.038348    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.038483    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.041735    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.043311    2581 server.go:1019] "Starting client certificate rotation"
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.043416    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 17:59:35.045582 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.043446    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 17:59:35.046607 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.046591    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 17:59:35.050018 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.049986    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 17:59:35.063469 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.063438    2581 log.go:25] "Validated CRI v1 runtime API"
Aug 04 17:59:35.065715 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.065697    2581 log.go:25] "Validated CRI v1 image API"
Aug 04 17:59:35.068054 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.068019    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 04 17:59:35.071576 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.071551    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 89a69ff0-8c82-45b4-9f92-1700ae8efbba:/dev/nvme0n1p3 a95f3284-4641-408e-a00d-1a7eb3f179cc:/dev/nvme0n1p4]
Aug 04 17:59:35.071664 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.071572    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 04 17:59:35.074291 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.074272    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 17:59:35.076344 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.076225    2581 manager.go:216] Machine: {Timestamp:2026-08-04 17:59:35.075703817 +0000 UTC m=+0.420104965 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099788 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c874c27779d93d10fec59d578d164 SystemUUID:ec2c874c-2777-9d93-d10f-ec59d578d164 BootID:72e49d00-d334-467e-9c40-8f384cc1d5a3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bb:82:64:69:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bb:82:64:69:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:01:d9:1f:e6:b7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 04 17:59:35.076344 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.076341    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 04 17:59:35.076459 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.076427    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 04 17:59:35.077674 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.077647    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 04 17:59:35.077833 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.077678    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-212.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 04 17:59:35.078358 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.078346    2581 topology_manager.go:138] "Creating topology manager with none policy"
Aug 04 17:59:35.078393 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.078360    2581 container_manager_linux.go:309] "Creating device plugin manager"
Aug 04 17:59:35.078393 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.078367    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 17:59:35.078393 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.078381    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 17:59:35.079383 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.079370    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 04 17:59:35.083146 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.083125    2581 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 17:59:35.083296 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.083285    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 04 17:59:35.083326 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.083320    2581 kubelet.go:486] "Attempting to sync node with API server"
Aug 04 17:59:35.083353 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.083335    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 04 17:59:35.083379 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.083355    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 04 17:59:35.083379 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.083373    2581 kubelet.go:398] "Adding apiserver pod source"
Aug 04 17:59:35.083437 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.083385    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 04 17:59:35.083856 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.083843    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 17:59:35.083902 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.083864    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 17:59:35.086105 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.086087    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 04 17:59:35.087520 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.087505    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 04 17:59:35.087558 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.087538    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 04 17:59:35.089036 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089022    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 04 17:59:35.089036 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089055    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 04 17:59:35.089036 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089065    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 04 17:59:35.089186 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089071    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 04 17:59:35.089186 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089080    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 04 17:59:35.089186 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089085    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 04 17:59:35.089186 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089101    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 04 17:59:35.089186 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089106    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 04 17:59:35.089186 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089112    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 04 17:59:35.089186 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089118    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 04 17:59:35.089186 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089129    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 04 17:59:35.089186 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089135    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 04 17:59:35.089763 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089752    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 04 17:59:35.089763 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.089762    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 04 17:59:35.093318 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.093299    2581 server.go:1268] "Started kubelet"
Aug 04 17:59:35.093406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.093340    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 04 17:59:35.093585 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.093522    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 04 17:59:35.093714 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.093614    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 04 17:59:35.095705 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.095673    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 04 17:59:35.095943 ip-10-0-129-212 systemd[1]: Started Kubernetes Kubelet.
Aug 04 17:59:35.096752 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.096115    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-212.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 04 17:59:35.096752 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.096326    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-212.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 04 17:59:35.096752 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.096462    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 04 17:59:35.098392 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.098370    2581 server.go:310] "Adding debug handlers to kubelet server"
Aug 04 17:59:35.100679 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.100638    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 04 17:59:35.103844 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.102269    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-212.ec2.internal.18c8ac0197bf9559  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-212.ec2.internal,UID:ip-10-0-129-212.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-212.ec2.internal,},FirstTimestamp:2026-08-04 17:59:35.093261657 +0000 UTC m=+0.437662805,LastTimestamp:2026-08-04 17:59:35.093261657 +0000 UTC m=+0.437662805,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-212.ec2.internal,}"
Aug 04 17:59:35.104697 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.104683    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 04 17:59:35.104747 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.104736    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 04 17:59:35.104843 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.104822    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-212.ec2.internal\" not found"
Aug 04 17:59:35.105458 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.105436    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 04 17:59:35.105458 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.105444    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 04 17:59:35.105609 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.105470    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 04 17:59:35.105609 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.105564    2581 reconstruct.go:97] "Volume reconstruction finished"
Aug 04 17:59:35.105609 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.105573    2581 reconciler.go:29] "Reconciler: start to sync state"
Aug 04 17:59:35.105735 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.105717    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 04 17:59:35.105735 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.105728    2581 factory.go:55] Registering systemd factory
Aug 04 17:59:35.105815 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.105738    2581 factory.go:223] Registration of the systemd container factory successfully
Aug 04 17:59:35.105980 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.105967    2581 factory.go:153] Registering CRI-O factory
Aug 04 17:59:35.106027 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.105983    2581 factory.go:223] Registration of the crio container factory successfully
Aug 04 17:59:35.106027 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.106002    2581 factory.go:103] Registering Raw factory
Aug 04 17:59:35.106027 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.106011    2581 manager.go:1196] Started watching for new ooms in manager
Aug 04 17:59:35.106419 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.106405    2581 manager.go:349] Starting recovery of all containers
Aug 04 17:59:35.116379 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.116346    2581 manager.go:354] Recovery completed
Aug 04 17:59:35.116730 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.116700    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 04 17:59:35.116816 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.116722    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-212.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 04 17:59:35.121169 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.121149    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:35.123103 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.123082    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:35.123199 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.123116    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:35.123199 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.123134    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:35.124188 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.124173    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 04 17:59:35.124188 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.124188    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 04 17:59:35.124316 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.124206    2581 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 17:59:35.124316 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.124224    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:35.125009 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.124988    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:35.125136 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.125024    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:35.125136 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.125037    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:35.125751 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.125739    2581 policy_none.go:49] "None policy: Start"
Aug 04 17:59:35.125798 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.125765    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 04 17:59:35.125798 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.125780    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 04 17:59:35.126167 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.126085    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-212.ec2.internal.18c8ac019986e199  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-212.ec2.internal,UID:ip-10-0-129-212.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-212.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-212.ec2.internal,},FirstTimestamp:2026-08-04 17:59:35.123100057 +0000 UTC m=+0.467501205,LastTimestamp:2026-08-04 17:59:35.123100057 +0000 UTC m=+0.467501205,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-212.ec2.internal,}"
Aug 04 17:59:35.127064 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.127036    2581 policy_none.go:47] "Start"
Aug 04 17:59:35.127859 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.127836    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56j5t"
Aug 04 17:59:35.136438 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.136199    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56j5t"
Aug 04 17:59:35.136834 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.136735    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-212.ec2.internal.18c8ac01998740f5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-212.ec2.internal,UID:ip-10-0-129-212.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-212.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-212.ec2.internal,},FirstTimestamp:2026-08-04 17:59:35.123124469 +0000 UTC m=+0.467525616,LastTimestamp:2026-08-04 17:59:35.123124469 +0000 UTC m=+0.467525616,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-212.ec2.internal,}"
Aug 04 17:59:35.139351 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.139285    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-212.ec2.internal.18c8ac0199877523  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-212.ec2.internal,UID:ip-10-0-129-212.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-212.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-212.ec2.internal,},FirstTimestamp:2026-08-04 17:59:35.123137827 +0000 UTC m=+0.467538975,LastTimestamp:2026-08-04 17:59:35.123137827 +0000 UTC m=+0.467538975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-212.ec2.internal,}"
Aug 04 17:59:35.165774 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.165750    2581 manager.go:337] "Starting Device Plugin manager"
Aug 04 17:59:35.182666 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.165790    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 04 17:59:35.182666 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.165804    2581 server.go:87] "Starting device plugin registration server"
Aug 04 17:59:35.182666 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.166129    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 04 17:59:35.182666 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.166142    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 04 17:59:35.182666 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.166231    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 04 17:59:35.182666 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.166312    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 04 17:59:35.182666 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.166320    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 04 17:59:35.182666 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.166918    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 04 17:59:35.182666 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.166976    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-212.ec2.internal\" not found"
Aug 04 17:59:35.207309 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.207266    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 04 17:59:35.208744 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.208723    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 04 17:59:35.208840 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.208763    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 04 17:59:35.208840 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.208786    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 04 17:59:35.208840 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.208822    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 04 17:59:35.211504 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.211480    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:35.266598 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.266568    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:35.267711 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.267675    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:35.267826 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.267720    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:35.267826 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.267733    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:35.267826 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.267758    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.279765 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.279688    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.279765 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.279723    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-212.ec2.internal\": node \"ip-10-0-129-212.ec2.internal\" not found"
Aug 04 17:59:35.301015 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.300985    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-212.ec2.internal\" not found"
Aug 04 17:59:35.309371 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.309342    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-212.ec2.internal"]
Aug 04 17:59:35.309453 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.309417    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:35.310344 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.310329    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:35.310409 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.310360    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:35.310409 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.310376    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:35.311797 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.311784    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:35.311932 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.311907    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.311980 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.311938    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:35.313100 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.313081    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:35.313219 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.313115    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:35.313219 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.313084    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:35.313219 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.313158    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:35.313219 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.313126    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:35.313402 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.313176    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:35.314613 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.314599    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.314675 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.314627    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:35.315369 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.315353    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:35.315439 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.315384    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:35.315439 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.315394    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:35.336779 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.336745    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-212.ec2.internal\" not found" node="ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.340976 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.340955    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-212.ec2.internal\" not found" node="ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.402060 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.402007    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-212.ec2.internal\" not found"
Aug 04 17:59:35.502490 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.502462    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-212.ec2.internal\" not found"
Aug 04 17:59:35.506796 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.506776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63c878e202c2c14d8539c00ad6086f2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal\" (UID: \"63c878e202c2c14d8539c00ad6086f2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.506858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.506803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63c878e202c2c14d8539c00ad6086f2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal\" (UID: \"63c878e202c2c14d8539c00ad6086f2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.506858 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.506819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f98a59467503bad43221398828496826-config\") pod \"kube-apiserver-proxy-ip-10-0-129-212.ec2.internal\" (UID: \"f98a59467503bad43221398828496826\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.603324 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.603247    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-212.ec2.internal\" not found"
Aug 04 17:59:35.607645 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.607620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f98a59467503bad43221398828496826-config\") pod \"kube-apiserver-proxy-ip-10-0-129-212.ec2.internal\" (UID: \"f98a59467503bad43221398828496826\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.607774 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.607658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63c878e202c2c14d8539c00ad6086f2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal\" (UID: \"63c878e202c2c14d8539c00ad6086f2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.607774 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.607685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63c878e202c2c14d8539c00ad6086f2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal\" (UID: \"63c878e202c2c14d8539c00ad6086f2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.607774 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.607729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f98a59467503bad43221398828496826-config\") pod \"kube-apiserver-proxy-ip-10-0-129-212.ec2.internal\" (UID: \"f98a59467503bad43221398828496826\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.607774 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.607733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63c878e202c2c14d8539c00ad6086f2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal\" (UID: \"63c878e202c2c14d8539c00ad6086f2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.607774 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.607735    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63c878e202c2c14d8539c00ad6086f2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal\" (UID: \"63c878e202c2c14d8539c00ad6086f2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.641819 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.641771    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.643930 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:35.643911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:35.703690 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.703650    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-212.ec2.internal\" not found"
Aug 04 17:59:35.804213 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.804173    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-212.ec2.internal\" not found"
Aug 04 17:59:35.904772 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:35.904685    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-212.ec2.internal\" not found"
Aug 04 17:59:36.005181 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:36.005148    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-212.ec2.internal\" not found"
Aug 04 17:59:36.042589 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.042552    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 04 17:59:36.042978 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.042720    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 17:59:36.091299 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.091269    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:36.105181 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.105147    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:36.105181 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.105175    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 04 17:59:36.117497 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.117464    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 17:59:36.120450 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.120431    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 17:59:36.122635 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.122612    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-212.ec2.internal"
Aug 04 17:59:36.131122 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.131089    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 17:59:36.140662 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.140639    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4rl6p"
Aug 04 17:59:36.142760 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.142724    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 17:54:35 +0000 UTC" deadline="2028-04-30 19:09:52.313156098 +0000 UTC"
Aug 04 17:59:36.142806 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.142761    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15241h10m16.170398348s"
Aug 04 17:59:36.148587 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.148569    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4rl6p"
Aug 04 17:59:36.295730 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.295695    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:36.343397 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:36.343356    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c878e202c2c14d8539c00ad6086f2f.slice/crio-320c7b8f354691f918405d01ce821930628fa23779599a16778b1152b3374040 WatchSource:0}: Error finding container 320c7b8f354691f918405d01ce821930628fa23779599a16778b1152b3374040: Status 404 returned error can't find the container with id 320c7b8f354691f918405d01ce821930628fa23779599a16778b1152b3374040
Aug 04 17:59:36.343621 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:36.343598    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98a59467503bad43221398828496826.slice/crio-0895f06a745043f2d09e173cc2ffb417b40292f657807c72950918eb6d75c3d1 WatchSource:0}: Error finding container 0895f06a745043f2d09e173cc2ffb417b40292f657807c72950918eb6d75c3d1: Status 404 returned error can't find the container with id 0895f06a745043f2d09e173cc2ffb417b40292f657807c72950918eb6d75c3d1
Aug 04 17:59:36.348701 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.348679    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 17:59:36.376371 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.376343    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:36.898080 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:36.898036    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:37.084870 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.084836    2581 apiserver.go:52] "Watching apiserver"
Aug 04 17:59:37.092321 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.092289    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 04 17:59:37.092853 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.092775    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-ksxcz","kube-system/konnectivity-agent-7n9kl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9","openshift-cluster-node-tuning-operator/tuned-r25pv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal","openshift-multus/multus-additional-cni-plugins-c28cc","openshift-multus/network-metrics-daemon-gdpj9","kube-system/kube-apiserver-proxy-ip-10-0-129-212.ec2.internal","openshift-dns/node-resolver-nnr9j","openshift-image-registry/node-ca-gwkqx","openshift-multus/multus-crxfm","openshift-network-diagnostics/network-check-target-gfgc5","openshift-network-operator/iptables-alerter-d9mjp","openshift-ovn-kubernetes/ovnkube-node-k2wfn"]
Aug 04 17:59:37.094993 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.094966    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d9mjp"
Aug 04 17:59:37.097057 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.097027    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:37.097198 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.097136    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:37.098391 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.098369    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 04 17:59:37.098524 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.098492    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 04 17:59:37.098614 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.098553    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 04 17:59:37.098614 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.098379    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hgxqf\""
Aug 04 17:59:37.098953 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.098924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.100438 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.100417    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.101611 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.101296    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 04 17:59:37.101611 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.101427    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 04 17:59:37.101611 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.101474    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 04 17:59:37.101818 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.101718    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5qffw\""
Aug 04 17:59:37.102599 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.102022    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.102762 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.102740    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 04 17:59:37.103040 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.103021    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 04 17:59:37.103329 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.103308    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bj8l9\""
Aug 04 17:59:37.104489 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.104463    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 04 17:59:37.104972 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.104951    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 04 17:59:37.105296 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.105280    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 04 17:59:37.105454 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.105438    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 04 17:59:37.105669 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.105652    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-slbq2\""
Aug 04 17:59:37.105750 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.105735    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 04 17:59:37.106738 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.106437    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:37.106738 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.106522    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:37.110110 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.110088    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nnr9j"
Aug 04 17:59:37.111586 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.111566    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gwkqx"
Aug 04 17:59:37.122445 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.122404    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:37.122627 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.122463    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 04 17:59:37.122627 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.122517    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:37.122767 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.122739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.122987 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.122964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 17:59:37.123222 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123202    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 04 17:59:37.123294 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123232    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 04 17:59:37.123358 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-registration-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.123358 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-device-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.123504 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zd5f\" (UniqueName: \"kubernetes.io/projected/7bce4186-4d07-4d02-b6d7-3d7210b550c5-kube-api-access-8zd5f\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.123504 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dee28c57-e4ad-482a-96c4-5dcff576109d-tmp\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.123504 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:37.123504 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/56e86da9-f3b0-4819-872b-8f4944af85e8-kube-api-access-bb792\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:37.123504 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.123672 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-sys-fs\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.123672 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-tuned\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.123672 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-system-cni-dir\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.123672 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-kubelet-config\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:37.123819 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-modprobe-d\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.123819 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-os-release\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.123819 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:37.123921 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-sys\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.123921 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-var-lib-kubelet\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.123921 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x788\" (UniqueName: \"kubernetes.io/projected/442c48c7-0e14-4940-b1e1-ef2a4b053953-kube-api-access-9x788\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.124070 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.124070 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-socket-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.124070 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.123970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/442c48c7-0e14-4940-b1e1-ef2a4b053953-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.124070 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4stwh\" (UniqueName: \"kubernetes.io/projected/621b62d7-f809-4ede-b423-6044448b17c4-kube-api-access-4stwh\") pod \"iptables-alerter-d9mjp\" (UID: \"621b62d7-f809-4ede-b423-6044448b17c4\") " pod="openshift-network-operator/iptables-alerter-d9mjp"
Aug 04 17:59:37.124070 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-lib-modules\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.124376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/442c48c7-0e14-4940-b1e1-ef2a4b053953-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.124376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/621b62d7-f809-4ede-b423-6044448b17c4-host-slash\") pod \"iptables-alerter-d9mjp\" (UID: \"621b62d7-f809-4ede-b423-6044448b17c4\") " pod="openshift-network-operator/iptables-alerter-d9mjp"
Aug 04 17:59:37.124376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-kubernetes\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.124376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-sysctl-d\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.124376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-sysctl-conf\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.124376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-run\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.124376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll54c\" (UniqueName: \"kubernetes.io/projected/dee28c57-e4ad-482a-96c4-5dcff576109d-kube-api-access-ll54c\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.124376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-cnibin\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.124376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/442c48c7-0e14-4940-b1e1-ef2a4b053953-cni-binary-copy\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.124376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-dbus\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:37.124759 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124549    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 04 17:59:37.124799 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124758    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k7kdb\""
Aug 04 17:59:37.125005 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124947    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 04 17:59:37.125107 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.125015    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 04 17:59:37.125220 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.124149    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.125664 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.125644    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 04 17:59:37.125781 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.125763    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-48sr2\""
Aug 04 17:59:37.125868 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.125853    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lkvd2\""
Aug 04 17:59:37.125951 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.125937    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 04 17:59:37.126005 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.125948    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wj6rg\""
Aug 04 17:59:37.126005 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.125944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-sysconfig\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.126107 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.126032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-systemd\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.126107 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.126069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-host\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.126203 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.126108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.126423 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.126396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/621b62d7-f809-4ede-b423-6044448b17c4-iptables-alerter-script\") pod \"iptables-alerter-d9mjp\" (UID: \"621b62d7-f809-4ede-b423-6044448b17c4\") " pod="openshift-network-operator/iptables-alerter-d9mjp"
Aug 04 17:59:37.128282 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.128254    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 04 17:59:37.128455 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.128254    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nq5cc\""
Aug 04 17:59:37.128579 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.128495    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 04 17:59:37.128696 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.128530    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 04 17:59:37.128912 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.128889    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 04 17:59:37.128999 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.128963    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 04 17:59:37.129200 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.129180    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 04 17:59:37.150900 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.150793    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:54:36 +0000 UTC" deadline="2028-03-31 21:16:26.901388242 +0000 UTC"
Aug 04 17:59:37.150900 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.150854    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14523h16m49.750539208s"
Aug 04 17:59:37.206847 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.206821    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 04 17:59:37.214891 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.214796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal" event={"ID":"63c878e202c2c14d8539c00ad6086f2f","Type":"ContainerStarted","Data":"320c7b8f354691f918405d01ce821930628fa23779599a16778b1152b3374040"}
Aug 04 17:59:37.217025 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.216973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-212.ec2.internal" event={"ID":"f98a59467503bad43221398828496826","Type":"ContainerStarted","Data":"0895f06a745043f2d09e173cc2ffb417b40292f657807c72950918eb6d75c3d1"}
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-kubelet\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41812be4-3b11-4893-a17b-7a67d8938e03-konnectivity-ca\") pod \"konnectivity-agent-7n9kl\" (UID: \"41812be4-3b11-4893-a17b-7a67d8938e03\") " pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-sys-fs\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-system-cni-dir\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-kubelet-config\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-var-lib-cni-multus\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.226835    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-sys-fs\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.226942    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs podName:56e86da9-f3b0-4819-872b-8f4944af85e8 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:37.726897962 +0000 UTC m=+3.071299123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs") pod "network-metrics-daemon-gdpj9" (UID: "56e86da9-f3b0-4819-872b-8f4944af85e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/feb74e0b-0f63-47a7-8853-85ee79a22f73-env-overrides\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-system-cni-dir\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.226987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:37.227123 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8gbh\" (UniqueName: \"kubernetes.io/projected/883287f9-439b-406f-92be-57c88b121f72-kube-api-access-g8gbh\") pod \"node-resolver-nnr9j\" (UID: \"883287f9-439b-406f-92be-57c88b121f72\") " pod="openshift-dns/node-resolver-nnr9j"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-kubelet-config\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-var-lib-cni-bin\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-etc-openvswitch\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.227209    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.227249    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret podName:8626a82d-8bbc-4e96-bd2d-bf7a1ec40786 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:37.727235269 +0000 UTC m=+3.071636410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret") pod "global-pull-secret-syncer-ksxcz" (UID: "8626a82d-8bbc-4e96-bd2d-bf7a1ec40786") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6516e99e-3bc1-46a1-ad86-2f72e109c596-host\") pod \"node-ca-gwkqx\" (UID: \"6516e99e-3bc1-46a1-ad86-2f72e109c596\") " pod="openshift-image-registry/node-ca-gwkqx"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41812be4-3b11-4893-a17b-7a67d8938e03-agent-certs\") pod \"konnectivity-agent-7n9kl\" (UID: \"41812be4-3b11-4893-a17b-7a67d8938e03\") " pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-sys\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-var-lib-kubelet\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-cni-binary-copy\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-run-k8s-cni-cncf-io\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-sys\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4stwh\" (UniqueName: \"kubernetes.io/projected/621b62d7-f809-4ede-b423-6044448b17c4-kube-api-access-4stwh\") pod \"iptables-alerter-d9mjp\" (UID: \"621b62d7-f809-4ede-b423-6044448b17c4\") " pod="openshift-network-operator/iptables-alerter-d9mjp"
Aug 04 17:59:37.228060 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-lib-modules\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-run-openvswitch\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-sysctl-conf\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-dbus\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-run-ovn\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-var-lib-kubelet\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/feb74e0b-0f63-47a7-8853-85ee79a22f73-ovnkube-script-lib\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wts\" (UniqueName: \"kubernetes.io/projected/feb74e0b-0f63-47a7-8853-85ee79a22f73-kube-api-access-85wts\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-registration-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zd5f\" (UniqueName: \"kubernetes.io/projected/7bce4186-4d07-4d02-b6d7-3d7210b550c5-kube-api-access-8zd5f\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883287f9-439b-406f-92be-57c88b121f72-tmp-dir\") pod \"node-resolver-nnr9j\" (UID: \"883287f9-439b-406f-92be-57c88b121f72\") " pod="openshift-dns/node-resolver-nnr9j"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-run-netns\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-cni-bin\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s2wg\" (UniqueName: \"kubernetes.io/projected/6516e99e-3bc1-46a1-ad86-2f72e109c596-kube-api-access-7s2wg\") pod \"node-ca-gwkqx\" (UID: \"6516e99e-3bc1-46a1-ad86-2f72e109c596\") " pod="openshift-image-registry/node-ca-gwkqx"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.227976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/56e86da9-f3b0-4819-872b-8f4944af85e8-kube-api-access-bb792\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:37.228973 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-lib-modules\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-sysctl-conf\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-tuned\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/883287f9-439b-406f-92be-57c88b121f72-hosts-file\") pod \"node-resolver-nnr9j\" (UID: \"883287f9-439b-406f-92be-57c88b121f72\") " pod="openshift-dns/node-resolver-nnr9j"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-systemd-units\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-modprobe-d\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-os-release\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-var-lib-kubelet\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gwx5\" (UniqueName: \"kubernetes.io/projected/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-kube-api-access-4gwx5\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-dbus\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-log-socket\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x788\" (UniqueName: \"kubernetes.io/projected/442c48c7-0e14-4940-b1e1-ef2a4b053953-kube-api-access-9x788\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-hostroot\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f75kw\" (UniqueName: \"kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw\") pod \"network-check-target-gfgc5\" (UID: \"fdbe2137-0bf9-4a65-a12b-684dc3da0657\") " pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228645    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-slash\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-run-systemd\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-socket-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.229639 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/442c48c7-0e14-4940-b1e1-ef2a4b053953-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-cni-dir\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-socket-dir-parent\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.228622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-os-release\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-run-multus-certs\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/442c48c7-0e14-4940-b1e1-ef2a4b053953-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/621b62d7-f809-4ede-b423-6044448b17c4-host-slash\") pod \"iptables-alerter-d9mjp\" (UID: \"621b62d7-f809-4ede-b423-6044448b17c4\") " pod="openshift-network-operator/iptables-alerter-d9mjp"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-os-release\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-run-netns\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-node-log\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/feb74e0b-0f63-47a7-8853-85ee79a22f73-ovnkube-config\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-kubernetes\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-sysctl-d\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-run\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll54c\" (UniqueName: \"kubernetes.io/projected/dee28c57-e4ad-482a-96c4-5dcff576109d-kube-api-access-ll54c\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-cnibin\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/442c48c7-0e14-4940-b1e1-ef2a4b053953-cni-binary-copy\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.230235 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/621b62d7-f809-4ede-b423-6044448b17c4-iptables-alerter-script\") pod \"iptables-alerter-d9mjp\" (UID: \"621b62d7-f809-4ede-b423-6044448b17c4\") " pod="openshift-network-operator/iptables-alerter-d9mjp"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6516e99e-3bc1-46a1-ad86-2f72e109c596-serviceca\") pod \"node-ca-gwkqx\" (UID: \"6516e99e-3bc1-46a1-ad86-2f72e109c596\") " pod="openshift-image-registry/node-ca-gwkqx"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-sysconfig\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-systemd\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-host\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-daemon-config\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-etc-kubernetes\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-var-lib-openvswitch\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229872    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-cni-netd\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/feb74e0b-0f63-47a7-8853-85ee79a22f73-ovn-node-metrics-cert\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.229967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-device-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dee28c57-e4ad-482a-96c4-5dcff576109d-tmp\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-system-cni-dir\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-cnibin\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-cnibin\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-systemd\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/621b62d7-f809-4ede-b423-6044448b17c4-host-slash\") pod \"iptables-alerter-d9mjp\" (UID: \"621b62d7-f809-4ede-b423-6044448b17c4\") " pod="openshift-network-operator/iptables-alerter-d9mjp"
Aug 04 17:59:37.231020 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-host\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-conf-dir\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-kubernetes\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-sysctl-d\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-socket-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/442c48c7-0e14-4940-b1e1-ef2a4b053953-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.231032    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/442c48c7-0e14-4940-b1e1-ef2a4b053953-cni-binary-copy\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.231087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-registration-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.231112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-run\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.231140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/442c48c7-0e14-4940-b1e1-ef2a4b053953-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.231177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/442c48c7-0e14-4940-b1e1-ef2a4b053953-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.231220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bce4186-4d07-4d02-b6d7-3d7210b550c5-device-dir\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.230297    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-sysconfig\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.231294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-modprobe-d\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.231946 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.231750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/621b62d7-f809-4ede-b423-6044448b17c4-iptables-alerter-script\") pod \"iptables-alerter-d9mjp\" (UID: \"621b62d7-f809-4ede-b423-6044448b17c4\") " pod="openshift-network-operator/iptables-alerter-d9mjp"
Aug 04 17:59:37.234657 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.234625    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dee28c57-e4ad-482a-96c4-5dcff576109d-tmp\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.234657 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.234645    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dee28c57-e4ad-482a-96c4-5dcff576109d-etc-tuned\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.247929 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.247894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll54c\" (UniqueName: \"kubernetes.io/projected/dee28c57-e4ad-482a-96c4-5dcff576109d-kube-api-access-ll54c\") pod \"tuned-r25pv\" (UID: \"dee28c57-e4ad-482a-96c4-5dcff576109d\") " pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.248605 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.248555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4stwh\" (UniqueName: \"kubernetes.io/projected/621b62d7-f809-4ede-b423-6044448b17c4-kube-api-access-4stwh\") pod \"iptables-alerter-d9mjp\" (UID: \"621b62d7-f809-4ede-b423-6044448b17c4\") " pod="openshift-network-operator/iptables-alerter-d9mjp"
Aug 04 17:59:37.248790 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.248745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/56e86da9-f3b0-4819-872b-8f4944af85e8-kube-api-access-bb792\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:37.249402 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.249352    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zd5f\" (UniqueName: \"kubernetes.io/projected/7bce4186-4d07-4d02-b6d7-3d7210b550c5-kube-api-access-8zd5f\") pod \"aws-ebs-csi-driver-node-zt9r9\" (UID: \"7bce4186-4d07-4d02-b6d7-3d7210b550c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.250856 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.250809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x788\" (UniqueName: \"kubernetes.io/projected/442c48c7-0e14-4940-b1e1-ef2a4b053953-kube-api-access-9x788\") pod \"multus-additional-cni-plugins-c28cc\" (UID: \"442c48c7-0e14-4940-b1e1-ef2a4b053953\") " pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.331164 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-run-ovn\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331164 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/feb74e0b-0f63-47a7-8853-85ee79a22f73-ovnkube-script-lib\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wts\" (UniqueName: \"kubernetes.io/projected/feb74e0b-0f63-47a7-8853-85ee79a22f73-kube-api-access-85wts\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883287f9-439b-406f-92be-57c88b121f72-tmp-dir\") pod \"node-resolver-nnr9j\" (UID: \"883287f9-439b-406f-92be-57c88b121f72\") " pod="openshift-dns/node-resolver-nnr9j"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-run-netns\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-cni-bin\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s2wg\" (UniqueName: \"kubernetes.io/projected/6516e99e-3bc1-46a1-ad86-2f72e109c596-kube-api-access-7s2wg\") pod \"node-ca-gwkqx\" (UID: \"6516e99e-3bc1-46a1-ad86-2f72e109c596\") " pod="openshift-image-registry/node-ca-gwkqx"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/883287f9-439b-406f-92be-57c88b121f72-hosts-file\") pod \"node-resolver-nnr9j\" (UID: \"883287f9-439b-406f-92be-57c88b121f72\") " pod="openshift-dns/node-resolver-nnr9j"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-systemd-units\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-var-lib-kubelet\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gwx5\" (UniqueName: \"kubernetes.io/projected/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-kube-api-access-4gwx5\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-log-socket\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-hostroot\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331422 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75kw\" (UniqueName: \"kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw\") pod \"network-check-target-gfgc5\" (UID: \"fdbe2137-0bf9-4a65-a12b-684dc3da0657\") " pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-slash\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-run-systemd\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-cni-dir\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-socket-dir-parent\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-run-multus-certs\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-os-release\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-run-netns\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-node-log\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/feb74e0b-0f63-47a7-8853-85ee79a22f73-ovnkube-config\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6516e99e-3bc1-46a1-ad86-2f72e109c596-serviceca\") pod \"node-ca-gwkqx\" (UID: \"6516e99e-3bc1-46a1-ad86-2f72e109c596\") " pod="openshift-image-registry/node-ca-gwkqx"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-daemon-config\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-etc-kubernetes\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-var-lib-openvswitch\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-cni-netd\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/feb74e0b-0f63-47a7-8853-85ee79a22f73-ovn-node-metrics-cert\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-system-cni-dir\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-cnibin\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.331873 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-conf-dir\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-kubelet\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41812be4-3b11-4893-a17b-7a67d8938e03-konnectivity-ca\") pod \"konnectivity-agent-7n9kl\" (UID: \"41812be4-3b11-4893-a17b-7a67d8938e03\") " pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-var-lib-cni-multus\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/feb74e0b-0f63-47a7-8853-85ee79a22f73-env-overrides\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8gbh\" (UniqueName: \"kubernetes.io/projected/883287f9-439b-406f-92be-57c88b121f72-kube-api-access-g8gbh\") pod \"node-resolver-nnr9j\" (UID: \"883287f9-439b-406f-92be-57c88b121f72\") " pod="openshift-dns/node-resolver-nnr9j"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-var-lib-cni-bin\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-etc-openvswitch\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6516e99e-3bc1-46a1-ad86-2f72e109c596-host\") pod \"node-ca-gwkqx\" (UID: \"6516e99e-3bc1-46a1-ad86-2f72e109c596\") " pod="openshift-image-registry/node-ca-gwkqx"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.331983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41812be4-3b11-4893-a17b-7a67d8938e03-agent-certs\") pod \"konnectivity-agent-7n9kl\" (UID: \"41812be4-3b11-4893-a17b-7a67d8938e03\") " pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-cni-binary-copy\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332025    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-run-k8s-cni-cncf-io\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-run-openvswitch\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-log-socket\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-run-netns\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-hostroot\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.332637 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/883287f9-439b-406f-92be-57c88b121f72-hosts-file\") pod \"node-resolver-nnr9j\" (UID: \"883287f9-439b-406f-92be-57c88b121f72\") " pod="openshift-dns/node-resolver-nnr9j"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-systemd-units\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-cni-bin\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-slash\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-run-ovn\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-run-systemd\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-var-lib-cni-multus\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-socket-dir-parent\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-run-multus-certs\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-etc-kubernetes\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-cni-dir\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-var-lib-openvswitch\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-os-release\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332787    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-var-lib-cni-bin\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-run-netns\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-cni-netd\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883287f9-439b-406f-92be-57c88b121f72-tmp-dir\") pod \"node-resolver-nnr9j\" (UID: \"883287f9-439b-406f-92be-57c88b121f72\") " pod="openshift-dns/node-resolver-nnr9j"
Aug 04 17:59:37.333584 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-daemon-config\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-run-k8s-cni-cncf-io\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.332980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6516e99e-3bc1-46a1-ad86-2f72e109c596-host\") pod \"node-ca-gwkqx\" (UID: \"6516e99e-3bc1-46a1-ad86-2f72e109c596\") " pod="openshift-image-registry/node-ca-gwkqx"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333010    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-multus-conf-dir\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-system-cni-dir\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-host-kubelet\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-cnibin\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/feb74e0b-0f63-47a7-8853-85ee79a22f73-ovnkube-script-lib\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-run-openvswitch\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-host-var-lib-kubelet\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-node-log\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb74e0b-0f63-47a7-8853-85ee79a22f73-etc-openvswitch\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/feb74e0b-0f63-47a7-8853-85ee79a22f73-ovnkube-config\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41812be4-3b11-4893-a17b-7a67d8938e03-konnectivity-ca\") pod \"konnectivity-agent-7n9kl\" (UID: \"41812be4-3b11-4893-a17b-7a67d8938e03\") " pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6516e99e-3bc1-46a1-ad86-2f72e109c596-serviceca\") pod \"node-ca-gwkqx\" (UID: \"6516e99e-3bc1-46a1-ad86-2f72e109c596\") " pod="openshift-image-registry/node-ca-gwkqx"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.333968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-cni-binary-copy\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.334406 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.334257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/feb74e0b-0f63-47a7-8853-85ee79a22f73-env-overrides\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.336032 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.336006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/feb74e0b-0f63-47a7-8853-85ee79a22f73-ovn-node-metrics-cert\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.336853 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.336827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41812be4-3b11-4893-a17b-7a67d8938e03-agent-certs\") pod \"konnectivity-agent-7n9kl\" (UID: \"41812be4-3b11-4893-a17b-7a67d8938e03\") " pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 17:59:37.343477 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.343403    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:37.343477 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.343476    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:37.343719 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.343494    2581 projected.go:196] Error preparing data for projected volume kube-api-access-f75kw for pod openshift-network-diagnostics/network-check-target-gfgc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:37.343719 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.343573    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw podName:fdbe2137-0bf9-4a65-a12b-684dc3da0657 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:37.843548801 +0000 UTC m=+3.187949959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f75kw" (UniqueName: "kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw") pod "network-check-target-gfgc5" (UID: "fdbe2137-0bf9-4a65-a12b-684dc3da0657") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:37.345714 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.345659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gwx5\" (UniqueName: \"kubernetes.io/projected/9faf7e40-0a3c-472f-8bbd-2897f7bcb23e-kube-api-access-4gwx5\") pod \"multus-crxfm\" (UID: \"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e\") " pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.345714 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.345660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wts\" (UniqueName: \"kubernetes.io/projected/feb74e0b-0f63-47a7-8853-85ee79a22f73-kube-api-access-85wts\") pod \"ovnkube-node-k2wfn\" (UID: \"feb74e0b-0f63-47a7-8853-85ee79a22f73\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.345950 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.345846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s2wg\" (UniqueName: \"kubernetes.io/projected/6516e99e-3bc1-46a1-ad86-2f72e109c596-kube-api-access-7s2wg\") pod \"node-ca-gwkqx\" (UID: \"6516e99e-3bc1-46a1-ad86-2f72e109c596\") " pod="openshift-image-registry/node-ca-gwkqx"
Aug 04 17:59:37.347158 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.347101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8gbh\" (UniqueName: \"kubernetes.io/projected/883287f9-439b-406f-92be-57c88b121f72-kube-api-access-g8gbh\") pod \"node-resolver-nnr9j\" (UID: \"883287f9-439b-406f-92be-57c88b121f72\") " pod="openshift-dns/node-resolver-nnr9j"
Aug 04 17:59:37.424146 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.424024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d9mjp"
Aug 04 17:59:37.440398 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.440360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9"
Aug 04 17:59:37.452106 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.452044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r25pv"
Aug 04 17:59:37.460721 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.460687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c28cc"
Aug 04 17:59:37.468891 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.468611    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nnr9j"
Aug 04 17:59:37.477277 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.477247    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gwkqx"
Aug 04 17:59:37.487531 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.487490    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-crxfm"
Aug 04 17:59:37.496099 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.496064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 17:59:37.503795 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.503754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 17:59:37.735988 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.735891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:37.735988 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.735939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:37.736246 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.736099    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:37.736246 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.736123    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:37.736246 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.736177    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret podName:8626a82d-8bbc-4e96-bd2d-bf7a1ec40786 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:38.736162539 +0000 UTC m=+4.080563676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret") pod "global-pull-secret-syncer-ksxcz" (UID: "8626a82d-8bbc-4e96-bd2d-bf7a1ec40786") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:37.736246 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.736193    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs podName:56e86da9-f3b0-4819-872b-8f4944af85e8 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:38.736185166 +0000 UTC m=+4.080586303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs") pod "network-metrics-daemon-gdpj9" (UID: "56e86da9-f3b0-4819-872b-8f4944af85e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:37.938261 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:37.938226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75kw\" (UniqueName: \"kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw\") pod \"network-check-target-gfgc5\" (UID: \"fdbe2137-0bf9-4a65-a12b-684dc3da0657\") " pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:37.938420 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.938395    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:37.938420 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.938416    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:37.938501 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.938427    2581 projected.go:196] Error preparing data for projected volume kube-api-access-f75kw for pod openshift-network-diagnostics/network-check-target-gfgc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:37.938501 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:37.938488    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw podName:fdbe2137-0bf9-4a65-a12b-684dc3da0657 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:38.938471625 +0000 UTC m=+4.282872769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f75kw" (UniqueName: "kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw") pod "network-check-target-gfgc5" (UID: "fdbe2137-0bf9-4a65-a12b-684dc3da0657") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:38.151222 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:38.151106    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:54:36 +0000 UTC" deadline="2028-02-27 06:52:45.208830724 +0000 UTC"
Aug 04 17:59:38.151222 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:38.151167    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13716h53m7.057668103s"
Aug 04 17:59:38.505188 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:38.505119    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883287f9_439b_406f_92be_57c88b121f72.slice/crio-969907ec81fcf148bd4fcc7225e81c3f7728aaefe0afa3381f11e20bff464eed WatchSource:0}: Error finding container 969907ec81fcf148bd4fcc7225e81c3f7728aaefe0afa3381f11e20bff464eed: Status 404 returned error can't find the container with id 969907ec81fcf148bd4fcc7225e81c3f7728aaefe0afa3381f11e20bff464eed
Aug 04 17:59:38.507679 ip-10-0-129-212 kubenswrapper[2581]: W0804 17:59:38.507647    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee28c57_e4ad_482a_96c4_5dcff576109d.slice/crio-998b60d6631cda944aa94326ad0922c82db62a0eb4c898127b54c1df907edd56 WatchSource:0}: Error finding container 998b60d6631cda944aa94326ad0922c82db62a0eb4c898127b54c1df907edd56: Status 404 returned error can't find the container with id 998b60d6631cda944aa94326ad0922c82db62a0eb4c898127b54c1df907edd56
Aug 04 17:59:38.745340 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:38.745029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:38.745340 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:38.745240    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:38.745340 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:38.745281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:38.745340 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:38.745310    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs podName:56e86da9-f3b0-4819-872b-8f4944af85e8 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:40.74529383 +0000 UTC m=+6.089694971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs") pod "network-metrics-daemon-gdpj9" (UID: "56e86da9-f3b0-4819-872b-8f4944af85e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:38.745669 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:38.745379    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:38.745669 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:38.745441    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret podName:8626a82d-8bbc-4e96-bd2d-bf7a1ec40786 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:40.745426841 +0000 UTC m=+6.089827978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret") pod "global-pull-secret-syncer-ksxcz" (UID: "8626a82d-8bbc-4e96-bd2d-bf7a1ec40786") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:38.946777 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:38.946732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75kw\" (UniqueName: \"kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw\") pod \"network-check-target-gfgc5\" (UID: \"fdbe2137-0bf9-4a65-a12b-684dc3da0657\") " pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:38.946963 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:38.946929    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:38.946963 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:38.946962    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:38.947054 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:38.946976    2581 projected.go:196] Error preparing data for projected volume kube-api-access-f75kw for pod openshift-network-diagnostics/network-check-target-gfgc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:38.947054 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:38.947037    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw podName:fdbe2137-0bf9-4a65-a12b-684dc3da0657 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:40.947022034 +0000 UTC m=+6.291423169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f75kw" (UniqueName: "kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw") pod "network-check-target-gfgc5" (UID: "fdbe2137-0bf9-4a65-a12b-684dc3da0657") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:39.210958 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.210299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:39.210958 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:39.210443    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:39.210958 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.210312    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:39.210958 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:39.210563    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:39.210958 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.210299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:39.210958 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:39.210643    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:39.224931 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.224887    2581 generic.go:356] "Generic (PLEG): container finished" podID="63c878e202c2c14d8539c00ad6086f2f" containerID="63b24fbfb3fa556b0c8438a04c856b500964cc4c86072f8cb1d5a00952a70382" exitCode=0
Aug 04 17:59:39.225174 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.225015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal" event={"ID":"63c878e202c2c14d8539c00ad6086f2f","Type":"ContainerDied","Data":"63b24fbfb3fa556b0c8438a04c856b500964cc4c86072f8cb1d5a00952a70382"}
Aug 04 17:59:39.232239 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.232104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c28cc" event={"ID":"442c48c7-0e14-4940-b1e1-ef2a4b053953","Type":"ContainerStarted","Data":"394d2568af0ebb1b47d9e62dc670084ef9a8076d2d87a236e067db248d247552"}
Aug 04 17:59:39.238169 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.238134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gwkqx" event={"ID":"6516e99e-3bc1-46a1-ad86-2f72e109c596","Type":"ContainerStarted","Data":"5cb2fdcfce73e1f820def9e7c7983884bef43e0a9a076f08fdf9c7122e799d47"}
Aug 04 17:59:39.241428 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.241392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-crxfm" event={"ID":"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e","Type":"ContainerStarted","Data":"b7ad31da4a3d6de09e31a1ea94c5e0e13f9f3c617bedd4cacd4d8b27f86c6a07"}
Aug 04 17:59:39.260161 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.256674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9" event={"ID":"7bce4186-4d07-4d02-b6d7-3d7210b550c5","Type":"ContainerStarted","Data":"94ca9d8c2f103529642d6ed5905d71d1d43da79316de9699b5315eaebb6da87d"}
Aug 04 17:59:39.269624 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.269581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d9mjp" event={"ID":"621b62d7-f809-4ede-b423-6044448b17c4","Type":"ContainerStarted","Data":"c46ce9824e0fcaca544001d6173828ab941467fe56ffbc24724f5eefbb606378"}
Aug 04 17:59:39.275442 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.275401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn" event={"ID":"feb74e0b-0f63-47a7-8853-85ee79a22f73","Type":"ContainerStarted","Data":"7add92bb73e93914dd5b7c2dd926de5ae5066a33e7db452a4026803c5739d02e"}
Aug 04 17:59:39.280368 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.280330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r25pv" event={"ID":"dee28c57-e4ad-482a-96c4-5dcff576109d","Type":"ContainerStarted","Data":"998b60d6631cda944aa94326ad0922c82db62a0eb4c898127b54c1df907edd56"}
Aug 04 17:59:39.286893 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.286850    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-212.ec2.internal" event={"ID":"f98a59467503bad43221398828496826","Type":"ContainerStarted","Data":"d59a194b29e15c6df2dc9fb2c85ff3a9006d4391afc02546a375804d114423de"}
Aug 04 17:59:39.290140 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.290103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7n9kl" event={"ID":"41812be4-3b11-4893-a17b-7a67d8938e03","Type":"ContainerStarted","Data":"4c9255893d7945044f06abc746b945c1ab5d0ad4d7d5cee40921b96b953172f4"}
Aug 04 17:59:39.294469 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:39.294435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nnr9j" event={"ID":"883287f9-439b-406f-92be-57c88b121f72","Type":"ContainerStarted","Data":"969907ec81fcf148bd4fcc7225e81c3f7728aaefe0afa3381f11e20bff464eed"}
Aug 04 17:59:40.326436 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:40.326387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal" event={"ID":"63c878e202c2c14d8539c00ad6086f2f","Type":"ContainerStarted","Data":"8114376b05925c0f33991a4ba24946c9a44984b05d1a0bb3d749c9d3d71952a4"}
Aug 04 17:59:40.346099 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:40.346029    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-212.ec2.internal" podStartSLOduration=4.34600514 podStartE2EDuration="4.34600514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:39.307033134 +0000 UTC m=+4.651434294" watchObservedRunningTime="2026-08-04 17:59:40.34600514 +0000 UTC m=+5.690406302"
Aug 04 17:59:40.761763 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:40.761655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:40.761763 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:40.761713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:40.762030 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:40.761856    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:40.762030 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:40.761937    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret podName:8626a82d-8bbc-4e96-bd2d-bf7a1ec40786 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:44.761916039 +0000 UTC m=+10.106317181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret") pod "global-pull-secret-syncer-ksxcz" (UID: "8626a82d-8bbc-4e96-bd2d-bf7a1ec40786") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:40.762446 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:40.762423    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:40.762541 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:40.762487    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs podName:56e86da9-f3b0-4819-872b-8f4944af85e8 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:44.762469597 +0000 UTC m=+10.106870743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs") pod "network-metrics-daemon-gdpj9" (UID: "56e86da9-f3b0-4819-872b-8f4944af85e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:40.964295 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:40.964252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75kw\" (UniqueName: \"kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw\") pod \"network-check-target-gfgc5\" (UID: \"fdbe2137-0bf9-4a65-a12b-684dc3da0657\") " pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:40.964490 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:40.964431    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:40.964490 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:40.964454    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:40.964490 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:40.964467    2581 projected.go:196] Error preparing data for projected volume kube-api-access-f75kw for pod openshift-network-diagnostics/network-check-target-gfgc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:40.964687 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:40.964531    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw podName:fdbe2137-0bf9-4a65-a12b-684dc3da0657 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:44.964511003 +0000 UTC m=+10.308912159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f75kw" (UniqueName: "kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw") pod "network-check-target-gfgc5" (UID: "fdbe2137-0bf9-4a65-a12b-684dc3da0657") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:41.212172 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:41.212060    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:41.212337 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:41.212220    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:41.212337 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:41.212308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:41.212474 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:41.212403    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:41.212474 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:41.212469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:41.212581 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:41.212544    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:43.210173 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:43.210135    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:43.210703 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:43.210286    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:43.210782 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:43.210711    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:43.210873 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:43.210849    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:43.211035 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:43.210999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:43.211147 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:43.211116    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:44.795332 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:44.795276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:44.795893 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:44.795351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:44.795893 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:44.795411    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:44.795893 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:44.795482    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs podName:56e86da9-f3b0-4819-872b-8f4944af85e8 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:52.795460412 +0000 UTC m=+18.139861549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs") pod "network-metrics-daemon-gdpj9" (UID: "56e86da9-f3b0-4819-872b-8f4944af85e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:44.795893 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:44.795522    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:44.795893 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:44.795598    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret podName:8626a82d-8bbc-4e96-bd2d-bf7a1ec40786 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:52.795578073 +0000 UTC m=+18.139979212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret") pod "global-pull-secret-syncer-ksxcz" (UID: "8626a82d-8bbc-4e96-bd2d-bf7a1ec40786") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:44.996929 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:44.996890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75kw\" (UniqueName: \"kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw\") pod \"network-check-target-gfgc5\" (UID: \"fdbe2137-0bf9-4a65-a12b-684dc3da0657\") " pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:44.997143 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:44.997112    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:44.997259 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:44.997146    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:44.997259 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:44.997162    2581 projected.go:196] Error preparing data for projected volume kube-api-access-f75kw for pod openshift-network-diagnostics/network-check-target-gfgc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:44.997259 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:44.997228    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw podName:fdbe2137-0bf9-4a65-a12b-684dc3da0657 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:52.99720663 +0000 UTC m=+18.341607794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f75kw" (UniqueName: "kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw") pod "network-check-target-gfgc5" (UID: "fdbe2137-0bf9-4a65-a12b-684dc3da0657") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:45.210402 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:45.210313    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:45.210568 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:45.210431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:45.210568 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:45.210444    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:45.210568 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:45.210517    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:45.210568 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:45.210542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:45.210803 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:45.210664    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:47.210661 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:47.210165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:47.210661 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:47.210207    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:47.210661 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:47.210296    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:47.210661 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:47.210167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:47.210661 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:47.210417    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:47.210661 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:47.210493    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:49.209935 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:49.209892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:49.210376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:49.209892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:49.210376 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:49.210058    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:49.210376 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:49.210036    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:49.210376 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:49.210152    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:49.210376 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:49.210233    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:51.209940 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:51.209901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:51.210447 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:51.209901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:51.210447 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:51.210072    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:51.210447 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:51.209901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:51.210447 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:51.210102    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:51.210447 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:51.210227    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:52.859269 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:52.859232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:52.859269 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:52.859271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:52.859773 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:52.859392    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:52.859773 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:52.859455    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs podName:56e86da9-f3b0-4819-872b-8f4944af85e8 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:08.859438334 +0000 UTC m=+34.203839470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs") pod "network-metrics-daemon-gdpj9" (UID: "56e86da9-f3b0-4819-872b-8f4944af85e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:52.859773 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:52.859398    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:52.859773 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:52.859540    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret podName:8626a82d-8bbc-4e96-bd2d-bf7a1ec40786 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:08.859526094 +0000 UTC m=+34.203927243 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret") pod "global-pull-secret-syncer-ksxcz" (UID: "8626a82d-8bbc-4e96-bd2d-bf7a1ec40786") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:53.060291 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:53.060256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75kw\" (UniqueName: \"kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw\") pod \"network-check-target-gfgc5\" (UID: \"fdbe2137-0bf9-4a65-a12b-684dc3da0657\") " pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:53.060471 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:53.060404    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:53.060471 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:53.060425    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:53.060471 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:53.060437    2581 projected.go:196] Error preparing data for projected volume kube-api-access-f75kw for pod openshift-network-diagnostics/network-check-target-gfgc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:53.060622 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:53.060499    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw podName:fdbe2137-0bf9-4a65-a12b-684dc3da0657 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:09.060480485 +0000 UTC m=+34.404881624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f75kw" (UniqueName: "kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw") pod "network-check-target-gfgc5" (UID: "fdbe2137-0bf9-4a65-a12b-684dc3da0657") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:53.210137 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:53.210035    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:53.210137 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:53.210104    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:53.210329 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:53.210066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:53.210329 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:53.210189    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:53.210329 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:53.210266    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:53.210329 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:53.210317    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:55.209911 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:55.209870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:55.210427 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:55.209994    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:55.210427 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:55.210006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:55.210427 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:55.210022    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:55.210427 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:55.210127    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:55.210427 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:55.210194    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:57.209182 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:57.209140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:57.209728 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:57.209143    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:57.209728 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:57.209272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:57.209728 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:57.209269    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:57.209728 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:57.209399    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:57.209728 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:57.209496    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:58.390238 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.389908    2581 generic.go:356] "Generic (PLEG): container finished" podID="442c48c7-0e14-4940-b1e1-ef2a4b053953" containerID="0355328ffb425c548e43c2562caf979c4082ebf521b2a9ec8a3af913bfe76d61" exitCode=0
Aug 04 17:59:58.390954 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.389985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c28cc" event={"ID":"442c48c7-0e14-4940-b1e1-ef2a4b053953","Type":"ContainerDied","Data":"0355328ffb425c548e43c2562caf979c4082ebf521b2a9ec8a3af913bfe76d61"}
Aug 04 17:59:58.391874 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.391850    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gwkqx" event={"ID":"6516e99e-3bc1-46a1-ad86-2f72e109c596","Type":"ContainerStarted","Data":"37b5eff12adbb44e39076cc4d0fe713919c641b73b547cffc5f7b386a2c7524e"}
Aug 04 17:59:58.393414 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.393388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-crxfm" event={"ID":"9faf7e40-0a3c-472f-8bbd-2897f7bcb23e","Type":"ContainerStarted","Data":"d2919ec284de13069b1ace3ce9c3c018341d7d0359cdafac47e648659c172a20"}
Aug 04 17:59:58.394748 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.394728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9" event={"ID":"7bce4186-4d07-4d02-b6d7-3d7210b550c5","Type":"ContainerStarted","Data":"6d174ef10f76a0dd9dc54c7e9789bf0b23ec57f2154223da06ccd39ce58c3988"}
Aug 04 17:59:58.396599 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.396581    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 17:59:58.396896 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.396874    2581 generic.go:356] "Generic (PLEG): container finished" podID="feb74e0b-0f63-47a7-8853-85ee79a22f73" containerID="bdb32548172133a143627299332da00ea573213f103f0af6b8b4caaf6677eb30" exitCode=1
Aug 04 17:59:58.396992 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.396931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn" event={"ID":"feb74e0b-0f63-47a7-8853-85ee79a22f73","Type":"ContainerStarted","Data":"143835db301f7014ab37b4d192324e5d3e5ccb21dda5ea502b31bd73159574e6"}
Aug 04 17:59:58.397033 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.396994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn" event={"ID":"feb74e0b-0f63-47a7-8853-85ee79a22f73","Type":"ContainerDied","Data":"bdb32548172133a143627299332da00ea573213f103f0af6b8b4caaf6677eb30"}
Aug 04 17:59:58.397130 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.397065    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn" event={"ID":"feb74e0b-0f63-47a7-8853-85ee79a22f73","Type":"ContainerStarted","Data":"27fa8076af71d928353a3dbcdd3c670c8aca0606097f5e8d66ce5995baf1c1c1"}
Aug 04 17:59:58.398384 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.398366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r25pv" event={"ID":"dee28c57-e4ad-482a-96c4-5dcff576109d","Type":"ContainerStarted","Data":"7e635ca1b888799c4b46029a2ba09dc8290aee49ef44d021101c6e26a52ac2a5"}
Aug 04 17:59:58.404692 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.404656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7n9kl" event={"ID":"41812be4-3b11-4893-a17b-7a67d8938e03","Type":"ContainerStarted","Data":"0a9af32916aba80a1299ef6ee262929522131d642aed2a5374cefa0d1700a619"}
Aug 04 17:59:58.406840 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.406811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nnr9j" event={"ID":"883287f9-439b-406f-92be-57c88b121f72","Type":"ContainerStarted","Data":"e0651447f5976e7836f2b64170e23148067aae31832e112767e04846dfebd0fa"}
Aug 04 17:59:58.416340 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.416287    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-212.ec2.internal" podStartSLOduration=22.416273137 podStartE2EDuration="22.416273137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:40.34648507 +0000 UTC m=+5.690886232" watchObservedRunningTime="2026-08-04 17:59:58.416273137 +0000 UTC m=+23.760674293"
Aug 04 17:59:58.432341 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.432297    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r25pv" podStartSLOduration=3.26784507 podStartE2EDuration="22.432280486s" totalImagesPullingTime="19.164435416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:58.43162728 +0000 UTC m=+23.776028437" watchObservedRunningTime="2026-08-04 17:59:58.432280486 +0000 UTC m=+23.776681643"
Aug 04 17:59:58.446839 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.446744    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nnr9j" podStartSLOduration=3.280443871 podStartE2EDuration="22.446728415s" totalImagesPullingTime="19.166284544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:58.446615139 +0000 UTC m=+23.791016295" watchObservedRunningTime="2026-08-04 17:59:58.446728415 +0000 UTC m=+23.791129572"
Aug 04 17:59:58.461774 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.461726    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gwkqx" podStartSLOduration=3.310239488 podStartE2EDuration="22.461709193s" totalImagesPullingTime="19.151469705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:58.461681541 +0000 UTC m=+23.806082697" watchObservedRunningTime="2026-08-04 17:59:58.461709193 +0000 UTC m=+23.806110343"
Aug 04 17:59:58.552408 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.552350    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-crxfm" podStartSLOduration=3.384412256 podStartE2EDuration="22.552333727s" totalImagesPullingTime="19.167921471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:58.519662618 +0000 UTC m=+23.864063843" watchObservedRunningTime="2026-08-04 17:59:58.552333727 +0000 UTC m=+23.896734883"
Aug 04 17:59:58.552667 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:58.552647    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7n9kl" podStartSLOduration=11.585686112 podStartE2EDuration="22.552643377s" totalImagesPullingTime="10.966957265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:58.552474387 +0000 UTC m=+23.896875525" watchObservedRunningTime="2026-08-04 17:59:58.552643377 +0000 UTC m=+23.897044533"
Aug 04 17:59:59.201442 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:59.201417    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 04 17:59:59.209624 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:59.209598    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 17:59:59.209789 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:59.209604    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 17:59:59.209789 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:59.209704    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 17:59:59.209880 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:59.209798    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 17:59:59.209880 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:59.209607    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 17:59:59.209960 ip-10-0-129-212 kubenswrapper[2581]: E0804 17:59:59.209904    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 17:59:59.411293 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:59.411206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9" event={"ID":"7bce4186-4d07-4d02-b6d7-3d7210b550c5","Type":"ContainerStarted","Data":"12e06e3520244708f5ad9bf7c35b191c1bf3a7d6e1a653f271a8e8e616b4009d"}
Aug 04 17:59:59.412849 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:59.412819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d9mjp" event={"ID":"621b62d7-f809-4ede-b423-6044448b17c4","Type":"ContainerStarted","Data":"ed08c6a1ec3323f2c0cf570baea397a06ab399fc23c29d77e808005294a16c00"}
Aug 04 17:59:59.415890 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:59.415871    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 17:59:59.416335 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:59.416312    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn" event={"ID":"feb74e0b-0f63-47a7-8853-85ee79a22f73","Type":"ContainerStarted","Data":"ea767a403f742ffcff37071e9846110c99d77c3c196e707ec47c94ca2ad93d0f"}
Aug 04 17:59:59.416413 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:59.416388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn" event={"ID":"feb74e0b-0f63-47a7-8853-85ee79a22f73","Type":"ContainerStarted","Data":"f42effb72727ccb3baa418c018a83de5f57534bacfac83fcece7798d4d5c8757"}
Aug 04 17:59:59.416463 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:59.416438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn" event={"ID":"feb74e0b-0f63-47a7-8853-85ee79a22f73","Type":"ContainerStarted","Data":"b688b5c4b0f60daeaa6b5198d88052c72cd86d23b502cb1f4228596e8046cd8d"}
Aug 04 17:59:59.429713 ip-10-0-129-212 kubenswrapper[2581]: I0804 17:59:59.429661    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d9mjp" podStartSLOduration=5.275631925 podStartE2EDuration="24.429644488s" totalImagesPullingTime="19.154012563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:59.429439925 +0000 UTC m=+24.773841086" watchObservedRunningTime="2026-08-04 17:59:59.429644488 +0000 UTC m=+24.774045649"
Aug 04 18:00:00.182088 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:00.181951    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T17:59:59.201439237Z","UUID":"e911290a-9d2f-4b66-a755-701fe3a9f8be","Handler":null,"Name":"","Endpoint":""}
Aug 04 18:00:00.184090 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:00.184063    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 04 18:00:00.184244 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:00.184115    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 04 18:00:00.421289 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:00.421250    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9" event={"ID":"7bce4186-4d07-4d02-b6d7-3d7210b550c5","Type":"ContainerStarted","Data":"57d661563be4ff7085ab649b6e97f37e8bed35149b4a74f2a9b8ce7425901beb"}
Aug 04 18:00:00.441436 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:00.441332    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zt9r9" podStartSLOduration=3.117876255 podStartE2EDuration="24.441311664s" totalImagesPullingTime="21.323435409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:00.440437136 +0000 UTC m=+25.784838293" watchObservedRunningTime="2026-08-04 18:00:00.441311664 +0000 UTC m=+25.785712823"
Aug 04 18:00:00.956299 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:00.956210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 18:00:00.956923 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:00.956894    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 18:00:01.209661 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:01.209581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:01.209832 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:01.209581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:00:01.209832 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:01.209703    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 18:00:01.209832 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:01.209789    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 18:00:01.209832 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:01.209581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 18:00:01.210036 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:01.209873    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 18:00:01.427223 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:01.427184    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:00:01.427673 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:01.427596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn" event={"ID":"feb74e0b-0f63-47a7-8853-85ee79a22f73","Type":"ContainerStarted","Data":"250ed30d492bded37d05580a99df64e6485d320d1dbd75ffe546e5b86280ce6f"}
Aug 04 18:00:01.428072 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:01.428021    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 18:00:01.428514 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:01.428496    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7n9kl"
Aug 04 18:00:03.209520 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:03.209306    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:03.210371 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:03.209327    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:00:03.210371 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:03.209361    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 18:00:03.210371 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:03.209695    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 18:00:03.210371 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:03.209751    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 18:00:03.210371 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:03.209635    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 18:00:03.434820 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:03.434526    2581 generic.go:356] "Generic (PLEG): container finished" podID="442c48c7-0e14-4940-b1e1-ef2a4b053953" containerID="fbef2b3df44cad997f98200de312eda9550a3651ada2986a06c6fcc3873bc5ec" exitCode=0
Aug 04 18:00:03.434820 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:03.434590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c28cc" event={"ID":"442c48c7-0e14-4940-b1e1-ef2a4b053953","Type":"ContainerDied","Data":"fbef2b3df44cad997f98200de312eda9550a3651ada2986a06c6fcc3873bc5ec"}
Aug 04 18:00:03.438623 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:03.438606    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:00:03.439085 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:03.439059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn" event={"ID":"feb74e0b-0f63-47a7-8853-85ee79a22f73","Type":"ContainerStarted","Data":"325e4f73559f8ef837550cf8674917be1f5b53dc4c8accf866673d1a31150900"}
Aug 04 18:00:03.439476 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:03.439461    2581 scope.go:117] "RemoveContainer" containerID="bdb32548172133a143627299332da00ea573213f103f0af6b8b4caaf6677eb30"
Aug 04 18:00:04.444974 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:04.444946    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:00:04.445391 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:04.445305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn" event={"ID":"feb74e0b-0f63-47a7-8853-85ee79a22f73","Type":"ContainerStarted","Data":"7b970875d06afd2bf7627e2e3a3833bbd333b008ce6fc95edcb160ba54d6879c"}
Aug 04 18:00:04.445656 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:04.445633    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 18:00:04.445794 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:04.445714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 18:00:04.445794 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:04.445748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 18:00:04.462646 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:04.462618    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 18:00:04.462791 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:04.462781    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 18:00:04.476025 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:04.475976    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn" podStartSLOduration=9.266717282 podStartE2EDuration="28.475958696s" totalImagesPullingTime="19.209241414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:04.47465136 +0000 UTC m=+29.819052540" watchObservedRunningTime="2026-08-04 18:00:04.475958696 +0000 UTC m=+29.820359853"
Aug 04 18:00:05.210363 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:05.210270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:05.210363 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:05.210314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:00:05.210568 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:05.210390    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 18:00:05.210568 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:05.210400    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 18:00:05.210568 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:05.210472    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 18:00:05.210568 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:05.210539    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 18:00:05.313788 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:05.313725    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ksxcz"]
Aug 04 18:00:05.314891 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:05.314855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gfgc5"]
Aug 04 18:00:05.315407 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:05.315385    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gdpj9"]
Aug 04 18:00:05.450273 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:05.450242    2581 generic.go:356] "Generic (PLEG): container finished" podID="442c48c7-0e14-4940-b1e1-ef2a4b053953" containerID="fca36fc9589aa53a43cb8ff02539044fcafd0b3aca4d40d253fb7a6f87b728f1" exitCode=0
Aug 04 18:00:05.450729 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:05.450316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c28cc" event={"ID":"442c48c7-0e14-4940-b1e1-ef2a4b053953","Type":"ContainerDied","Data":"fca36fc9589aa53a43cb8ff02539044fcafd0b3aca4d40d253fb7a6f87b728f1"}
Aug 04 18:00:05.450729 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:05.450391    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 18:00:05.450729 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:05.450406    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:00:05.450729 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:05.450494    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 18:00:05.450729 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:05.450551    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 18:00:05.450729 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:05.450592    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:05.450729 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:05.450641    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 18:00:06.455413 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:06.455315    2581 generic.go:356] "Generic (PLEG): container finished" podID="442c48c7-0e14-4940-b1e1-ef2a4b053953" containerID="349907d7e5e4dc4f68c750f23d287874736018bcedada2af92b00754bc21315a" exitCode=0
Aug 04 18:00:06.455413 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:06.455392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c28cc" event={"ID":"442c48c7-0e14-4940-b1e1-ef2a4b053953","Type":"ContainerDied","Data":"349907d7e5e4dc4f68c750f23d287874736018bcedada2af92b00754bc21315a"}
Aug 04 18:00:07.213692 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:07.213276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 18:00:07.213692 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:07.213296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:07.213692 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:07.213401    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 18:00:07.213692 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:07.213459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:00:07.213692 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:07.213575    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 18:00:07.213692 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:07.213670    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 18:00:08.877728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:08.877504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:00:08.878314 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:08.877753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 18:00:08.878314 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:08.877686    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:08.878314 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:08.877872    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:08.878314 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:08.877895    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs podName:56e86da9-f3b0-4819-872b-8f4944af85e8 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:40.877870535 +0000 UTC m=+66.222271672 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs") pod "network-metrics-daemon-gdpj9" (UID: "56e86da9-f3b0-4819-872b-8f4944af85e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:08.878314 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:08.877923    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret podName:8626a82d-8bbc-4e96-bd2d-bf7a1ec40786 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:40.877907229 +0000 UTC m=+66.222308366 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret") pod "global-pull-secret-syncer-ksxcz" (UID: "8626a82d-8bbc-4e96-bd2d-bf7a1ec40786") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:09.079578 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:09.079535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75kw\" (UniqueName: \"kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw\") pod \"network-check-target-gfgc5\" (UID: \"fdbe2137-0bf9-4a65-a12b-684dc3da0657\") " pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:09.079766 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:09.079712    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:09.079766 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:09.079735    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:09.079766 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:09.079746    2581 projected.go:196] Error preparing data for projected volume kube-api-access-f75kw for pod openshift-network-diagnostics/network-check-target-gfgc5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:09.079882 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:09.079797    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw podName:fdbe2137-0bf9-4a65-a12b-684dc3da0657 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:41.079784475 +0000 UTC m=+66.424185612 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f75kw" (UniqueName: "kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw") pod "network-check-target-gfgc5" (UID: "fdbe2137-0bf9-4a65-a12b-684dc3da0657") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:09.209792 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:09.209757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:00:09.209973 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:09.209821    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 18:00:09.209973 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:09.209828    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:09.209973 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:09.209927    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxcz" podUID="8626a82d-8bbc-4e96-bd2d-bf7a1ec40786"
Aug 04 18:00:09.210162 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:09.210084    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdpj9" podUID="56e86da9-f3b0-4819-872b-8f4944af85e8"
Aug 04 18:00:09.210377 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:09.210213    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfgc5" podUID="fdbe2137-0bf9-4a65-a12b-684dc3da0657"
Aug 04 18:00:10.958279 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:10.958252    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-212.ec2.internal" event="NodeReady"
Aug 04 18:00:10.958843 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:10.958407    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 04 18:00:10.994436 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:10.994404    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-565b8c87c4-9gpcq"]
Aug 04 18:00:11.026383 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.026333    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5x45c"]
Aug 04 18:00:11.026610 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.026588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.028741 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.028720    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 04 18:00:11.028862 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.028778    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cp5kp\""
Aug 04 18:00:11.029037 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.029018    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 04 18:00:11.029178 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.029073    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 04 18:00:11.038841 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.038816    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 04 18:00:11.048343 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.048312    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8brzb"]
Aug 04 18:00:11.048511 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.048433    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.050786 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.050751    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 04 18:00:11.050786 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.050769    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 04 18:00:11.050965 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.050762    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g9rrt\""
Aug 04 18:00:11.066027 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.065990    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-565b8c87c4-9gpcq"]
Aug 04 18:00:11.066218 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.066036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8brzb"
Aug 04 18:00:11.066218 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.066075    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5x45c"]
Aug 04 18:00:11.066218 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.066097    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8brzb"]
Aug 04 18:00:11.068428 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.068407    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 04 18:00:11.068565 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.068488    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 04 18:00:11.068636 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.068598    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fmq84\""
Aug 04 18:00:11.068804 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.068768    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 04 18:00:11.193870 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.193836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.193870 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.193882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-ca-trust-extracted\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.194271 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.193973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-certificates\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.194271 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.194037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.194271 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.194076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh5df\" (UniqueName: \"kubernetes.io/projected/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-kube-api-access-wh5df\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.194271 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.194097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qqs9\" (UniqueName: \"kubernetes.io/projected/d80c332a-acaa-4ecb-b197-e69f0c4116d5-kube-api-access-8qqs9\") pod \"ingress-canary-8brzb\" (UID: \"d80c332a-acaa-4ecb-b197-e69f0c4116d5\") " pod="openshift-ingress-canary/ingress-canary-8brzb"
Aug 04 18:00:11.194271 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.194120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-trusted-ca\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.194271 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.194135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-installation-pull-secrets\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.194271 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.194156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7jv\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-kube-api-access-wx7jv\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.194271 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.194207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-tmp-dir\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.194271 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.194244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-image-registry-private-configuration\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.194271 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.194272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-bound-sa-token\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.194680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.194297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-config-volume\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.194680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.194320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d80c332a-acaa-4ecb-b197-e69f0c4116d5-cert\") pod \"ingress-canary-8brzb\" (UID: \"d80c332a-acaa-4ecb-b197-e69f0c4116d5\") " pod="openshift-ingress-canary/ingress-canary-8brzb"
Aug 04 18:00:11.195174 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.195070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:11.210058 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.209954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:00:11.210242 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.209957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 18:00:11.210417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.209957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:11.212932 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.212893    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:00:11.212932 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.212908    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:00:11.213149 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.212940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h457s\""
Aug 04 18:00:11.213149 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.212897    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:00:11.213149 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.213003    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5s4h\""
Aug 04 18:00:11.214073 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.213946    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:00:11.290290 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.290257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:11.295247 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.295247 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-ca-trust-extracted\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.295444 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-certificates\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.295444 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:11.295352    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:11.295444 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:11.295375    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b8c87c4-9gpcq: secret "image-registry-tls" not found
Aug 04 18:00:11.295444 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.295444 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh5df\" (UniqueName: \"kubernetes.io/projected/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-kube-api-access-wh5df\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.295444 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:11.295433    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls podName:ecebf12c-a9cc-44dd-bee5-d210c3209aa9 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:11.795414884 +0000 UTC m=+37.139816035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls") pod "image-registry-565b8c87c4-9gpcq" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9") : secret "image-registry-tls" not found
Aug 04 18:00:11.295662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qqs9\" (UniqueName: \"kubernetes.io/projected/d80c332a-acaa-4ecb-b197-e69f0c4116d5-kube-api-access-8qqs9\") pod \"ingress-canary-8brzb\" (UID: \"d80c332a-acaa-4ecb-b197-e69f0c4116d5\") " pod="openshift-ingress-canary/ingress-canary-8brzb"
Aug 04 18:00:11.295662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-trusted-ca\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.295662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-installation-pull-secrets\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.295662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7jv\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-kube-api-access-wx7jv\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.295662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-tmp-dir\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.295662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-image-registry-private-configuration\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.295662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-bound-sa-token\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.295662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-ca-trust-extracted\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.295662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-config-volume\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.295662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.295653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d80c332a-acaa-4ecb-b197-e69f0c4116d5-cert\") pod \"ingress-canary-8brzb\" (UID: \"d80c332a-acaa-4ecb-b197-e69f0c4116d5\") " pod="openshift-ingress-canary/ingress-canary-8brzb"
Aug 04 18:00:11.295999 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:11.295821    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:11.295999 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:11.295876    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls podName:22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:11.795861318 +0000 UTC m=+37.140262455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls") pod "dns-default-5x45c" (UID: "22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:11.296423 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.296232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-certificates\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.296677 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.296657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-tmp-dir\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.296846 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.296821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-config-volume\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.297146 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.297120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-trusted-ca\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.300376 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.300346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-installation-pull-secrets\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.300500 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.300397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d80c332a-acaa-4ecb-b197-e69f0c4116d5-cert\") pod \"ingress-canary-8brzb\" (UID: \"d80c332a-acaa-4ecb-b197-e69f0c4116d5\") " pod="openshift-ingress-canary/ingress-canary-8brzb"
Aug 04 18:00:11.305666 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.305636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh5df\" (UniqueName: \"kubernetes.io/projected/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-kube-api-access-wh5df\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.305840 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.305818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qqs9\" (UniqueName: \"kubernetes.io/projected/d80c332a-acaa-4ecb-b197-e69f0c4116d5-kube-api-access-8qqs9\") pod \"ingress-canary-8brzb\" (UID: \"d80c332a-acaa-4ecb-b197-e69f0c4116d5\") " pod="openshift-ingress-canary/ingress-canary-8brzb"
Aug 04 18:00:11.305879 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.305854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7jv\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-kube-api-access-wx7jv\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.305879 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.305869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-bound-sa-token\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.306481 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.306451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-image-registry-private-configuration\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.375418 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.375375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8brzb"
Aug 04 18:00:11.590830 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.590788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8brzb"]
Aug 04 18:00:11.637037 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:11.636848    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80c332a_acaa_4ecb_b197_e69f0c4116d5.slice/crio-92372803bcb2bb10bf72ec1b3307e46d0d368f144556e0c45c62f384e0abf845 WatchSource:0}: Error finding container 92372803bcb2bb10bf72ec1b3307e46d0d368f144556e0c45c62f384e0abf845: Status 404 returned error can't find the container with id 92372803bcb2bb10bf72ec1b3307e46d0d368f144556e0c45c62f384e0abf845
Aug 04 18:00:11.800087 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.800034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:11.800286 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:11.800101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:11.800286 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:11.800207    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:11.800286 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:11.800211    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:11.800442 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:11.800299    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls podName:22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:12.800277677 +0000 UTC m=+38.144678823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls") pod "dns-default-5x45c" (UID: "22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:11.800442 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:11.800223    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b8c87c4-9gpcq: secret "image-registry-tls" not found
Aug 04 18:00:11.800442 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:11.800365    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls podName:ecebf12c-a9cc-44dd-bee5-d210c3209aa9 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:12.800351372 +0000 UTC m=+38.144752510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls") pod "image-registry-565b8c87c4-9gpcq" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9") : secret "image-registry-tls" not found
Aug 04 18:00:12.471464 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:12.471422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8brzb" event={"ID":"d80c332a-acaa-4ecb-b197-e69f0c4116d5","Type":"ContainerStarted","Data":"92372803bcb2bb10bf72ec1b3307e46d0d368f144556e0c45c62f384e0abf845"}
Aug 04 18:00:12.474091 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:12.474060    2581 generic.go:356] "Generic (PLEG): container finished" podID="442c48c7-0e14-4940-b1e1-ef2a4b053953" containerID="ec2a231f6cc64b7c1c0bd39ad384d7c651c57567a060d4d9e693cecec4f79c01" exitCode=0
Aug 04 18:00:12.474248 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:12.474120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c28cc" event={"ID":"442c48c7-0e14-4940-b1e1-ef2a4b053953","Type":"ContainerDied","Data":"ec2a231f6cc64b7c1c0bd39ad384d7c651c57567a060d4d9e693cecec4f79c01"}
Aug 04 18:00:12.807697 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:12.807657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:12.807898 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:12.807724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:12.807898 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:12.807843    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:12.807898 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:12.807846    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:12.807898 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:12.807855    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b8c87c4-9gpcq: secret "image-registry-tls" not found
Aug 04 18:00:12.808123 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:12.807916    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls podName:22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:14.807895714 +0000 UTC m=+40.152296849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls") pod "dns-default-5x45c" (UID: "22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:12.808123 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:12.807946    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls podName:ecebf12c-a9cc-44dd-bee5-d210c3209aa9 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:14.80792897 +0000 UTC m=+40.152330109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls") pod "image-registry-565b8c87c4-9gpcq" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9") : secret "image-registry-tls" not found
Aug 04 18:00:13.480827 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:13.480785    2581 generic.go:356] "Generic (PLEG): container finished" podID="442c48c7-0e14-4940-b1e1-ef2a4b053953" containerID="e8ae80f3a4a38b754554f8eb1df2797ffcfeb600d0a4036a3f7bab0620887bd4" exitCode=0
Aug 04 18:00:13.481344 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:13.480870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c28cc" event={"ID":"442c48c7-0e14-4940-b1e1-ef2a4b053953","Type":"ContainerDied","Data":"e8ae80f3a4a38b754554f8eb1df2797ffcfeb600d0a4036a3f7bab0620887bd4"}
Aug 04 18:00:14.426231 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:14.426186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:14.487356 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:14.487316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c28cc" event={"ID":"442c48c7-0e14-4940-b1e1-ef2a4b053953","Type":"ContainerStarted","Data":"ee6592fd1c6a6f8c925521310dbb92df9705c4a9e31fa9c7308a9cabae1c56c9"}
Aug 04 18:00:14.511462 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:14.511402    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c28cc" podStartSLOduration=7.433405948 podStartE2EDuration="38.51138597s" totalImagesPullingTime="31.077980022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:14.510991097 +0000 UTC m=+39.855392254" watchObservedRunningTime="2026-08-04 18:00:14.51138597 +0000 UTC m=+39.855787154"
Aug 04 18:00:14.515663 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:14.515629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:14.636343 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:14.636308    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:14.749953 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:14.749776    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:14.825134 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:14.825100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:14.825311 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:14.825147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:14.825311 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:14.825239    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:14.825311 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:14.825246    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:14.825422 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:14.825252    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b8c87c4-9gpcq: secret "image-registry-tls" not found
Aug 04 18:00:14.825422 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:14.825355    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls podName:22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:18.825334899 +0000 UTC m=+44.169736042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls") pod "dns-default-5x45c" (UID: "22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:14.825422 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:14.825379    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls podName:ecebf12c-a9cc-44dd-bee5-d210c3209aa9 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:18.825368306 +0000 UTC m=+44.169769446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls") pod "image-registry-565b8c87c4-9gpcq" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9") : secret "image-registry-tls" not found
Aug 04 18:00:15.424179 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:15.424130    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:15.491021 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:15.490982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8brzb" event={"ID":"d80c332a-acaa-4ecb-b197-e69f0c4116d5","Type":"ContainerStarted","Data":"15a7269784ffe709b25e97cdb407fee64eab6d1072bc97b123056cb9d06f0b01"}
Aug 04 18:00:15.513590 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:15.513548    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:15.803974 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:15.803943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:16.544560 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:16.544517    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:16.646654 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:16.646622    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:18.358763 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:18.358732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:18.858773 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:18.858733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:18.858944 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:18.858810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:18.858944 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:18.858921    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:18.858944 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:18.858929    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:18.858944 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:18.858941    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b8c87c4-9gpcq: secret "image-registry-tls" not found
Aug 04 18:00:18.859104 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:18.858992    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls podName:ecebf12c-a9cc-44dd-bee5-d210c3209aa9 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:26.858977078 +0000 UTC m=+52.203378213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls") pod "image-registry-565b8c87c4-9gpcq" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9") : secret "image-registry-tls" not found
Aug 04 18:00:18.859104 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:18.859007    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls podName:22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:26.859000793 +0000 UTC m=+52.203401928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls") pod "dns-default-5x45c" (UID: "22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:19.133366 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:19.133275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:19.199232 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:19.199199    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:22.859208 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.859149    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8brzb" podStartSLOduration=9.879026835 podStartE2EDuration="12.859131731s" totalImagesPullingTime="2.980104896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:15.508506962 +0000 UTC m=+40.852908119" watchObservedRunningTime="2026-08-04 18:00:22.859131731 +0000 UTC m=+48.203532925"
Aug 04 18:00:22.859984 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.859964    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-qk8jv"]
Aug 04 18:00:22.904862 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.904826    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"]
Aug 04 18:00:22.905023 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.904955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-qk8jv"
Aug 04 18:00:22.907616 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.907591    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:22.907727 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.907652    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:22.907727 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.907652    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wvtfv\""
Aug 04 18:00:22.918562 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.918531    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"]
Aug 04 18:00:22.918716 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.918671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:22.921437 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.921412    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 04 18:00:22.922360 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.922337    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Aug 04 18:00:22.922417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.922367    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 04 18:00:22.922453 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.922369    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-flkwk\""
Aug 04 18:00:22.922603 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.922584    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Aug 04 18:00:22.938376 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.938334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-qk8jv"]
Aug 04 18:00:22.938376 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.938376    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"]
Aug 04 18:00:22.938376 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.938390    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"]
Aug 04 18:00:22.938596 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.938466    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"
Aug 04 18:00:22.942239 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.942197    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Aug 04 18:00:22.943765 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.943743    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Aug 04 18:00:22.944788 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.944760    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:22.945566 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.945540    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:22.946725 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.946709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6s7r4\""
Aug 04 18:00:22.958860 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.958830    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"]
Aug 04 18:00:22.975007 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.974931    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-zxjxn"]
Aug 04 18:00:22.975222 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.975071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:22.977664 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.977641    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Aug 04 18:00:22.977815 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.977695    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:22.978104 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.978083    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:22.978225 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.978160    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pdpmd\""
Aug 04 18:00:22.988532 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.988499    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-xgggh"]
Aug 04 18:00:22.989376 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.989352    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:22.992715 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.992691    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Aug 04 18:00:22.993394 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.993371    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 04 18:00:22.993818 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.993797    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Aug 04 18:00:22.994231 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.994216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ww8s9\""
Aug 04 18:00:22.997002 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:22.996981    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 04 18:00:23.007863 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.007836    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Aug 04 18:00:23.008644 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.008619    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-785c67bddb-zlhb4"]
Aug 04 18:00:23.008783 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.008743    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.013118 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.013088    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Aug 04 18:00:23.013118 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.013098    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:23.013345 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.013199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Aug 04 18:00:23.013345 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.013219    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:23.013428 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.013368    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-p9jmq\""
Aug 04 18:00:23.018812 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.018789    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Aug 04 18:00:23.028721 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.028688    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"]
Aug 04 18:00:23.028866 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.028816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.032123 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.031395    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Aug 04 18:00:23.032123 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.031432    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Aug 04 18:00:23.032123 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.031538    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Aug 04 18:00:23.032123 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.031878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Aug 04 18:00:23.032123 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.031886    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Aug 04 18:00:23.032123 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.031898    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7b75h\""
Aug 04 18:00:23.032123 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.032021    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Aug 04 18:00:23.050806 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.050762    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"]
Aug 04 18:00:23.050975 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.050833    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-zxjxn"]
Aug 04 18:00:23.050975 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.050849    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-xgggh"]
Aug 04 18:00:23.050975 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.050862    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-785c67bddb-zlhb4"]
Aug 04 18:00:23.050975 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.050878    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"]
Aug 04 18:00:23.050975 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.050896    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"]
Aug 04 18:00:23.050975 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.050915    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"
Aug 04 18:00:23.053376 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.053346    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:23.053535 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.053378    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:23.053535 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.053387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Aug 04 18:00:23.053535 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.053440    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-kvtz4\""
Aug 04 18:00:23.053535 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.053383    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Aug 04 18:00:23.074085 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.074018    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-vl78f"]
Aug 04 18:00:23.074261 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.074162    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:23.076587 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.076562    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-86gq6\""
Aug 04 18:00:23.076708 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.076562    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Aug 04 18:00:23.076708 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.076567    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Aug 04 18:00:23.090820 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.090788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s4r8f\" (UID: \"cd3e57c1-5ced-4a81-99f5-1021eb303059\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:23.090984 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.090848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e8f66e5-a4db-45ee-877a-de0246618be7-tmp\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.090984 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.090869    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4wl\" (UniqueName: \"kubernetes.io/projected/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-kube-api-access-7c4wl\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:23.090984 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.090957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e8f66e5-a4db-45ee-877a-de0246618be7-serving-cert\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.091110 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.091001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:23.091110 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.091061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e8f66e5-a4db-45ee-877a-de0246618be7-trusted-ca-bundle\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.091110 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.091094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e8f66e5-a4db-45ee-877a-de0246618be7-snapshots\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.091246 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.091118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lm6q\" (UniqueName: \"kubernetes.io/projected/7e8f66e5-a4db-45ee-877a-de0246618be7-kube-api-access-8lm6q\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.091246 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.091139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:23.091246 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.091200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4962250-4950-463c-92ee-b6d105f03d80-config\") pod \"service-ca-operator-784f97bcf6-pfvlz\" (UID: \"d4962250-4950-463c-92ee-b6d105f03d80\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"
Aug 04 18:00:23.091348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.091257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e8f66e5-a4db-45ee-877a-de0246618be7-service-ca-bundle\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.091348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.091274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4962250-4950-463c-92ee-b6d105f03d80-serving-cert\") pod \"service-ca-operator-784f97bcf6-pfvlz\" (UID: \"d4962250-4950-463c-92ee-b6d105f03d80\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"
Aug 04 18:00:23.091348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.091310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mph9n\" (UniqueName: \"kubernetes.io/projected/cd3e57c1-5ced-4a81-99f5-1021eb303059-kube-api-access-mph9n\") pod \"cluster-samples-operator-559cbf5745-s4r8f\" (UID: \"cd3e57c1-5ced-4a81-99f5-1021eb303059\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:23.091348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.091330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9kgh\" (UniqueName: \"kubernetes.io/projected/d5277a01-9455-4067-9d90-067385c2116b-kube-api-access-n9kgh\") pod \"volume-data-source-validator-5b8776669-qk8jv\" (UID: \"d5277a01-9455-4067-9d90-067385c2116b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-qk8jv"
Aug 04 18:00:23.091348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.091348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5n2h\" (UniqueName: \"kubernetes.io/projected/d4962250-4950-463c-92ee-b6d105f03d80-kube-api-access-t5n2h\") pod \"service-ca-operator-784f97bcf6-pfvlz\" (UID: \"d4962250-4950-463c-92ee-b6d105f03d80\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"
Aug 04 18:00:23.098330 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.098291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"]
Aug 04 18:00:23.098330 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.098332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-vl78f"]
Aug 04 18:00:23.098563 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.098433    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-vl78f"
Aug 04 18:00:23.100804 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.100784    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6db7h\""
Aug 04 18:00:23.176624 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.176537    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:23.191675 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.191636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4962250-4950-463c-92ee-b6d105f03d80-config\") pod \"service-ca-operator-784f97bcf6-pfvlz\" (UID: \"d4962250-4950-463c-92ee-b6d105f03d80\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"
Aug 04 18:00:23.191675 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.191677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e8f66e5-a4db-45ee-877a-de0246618be7-service-ca-bundle\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.191852 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.191697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4962250-4950-463c-92ee-b6d105f03d80-serving-cert\") pod \"service-ca-operator-784f97bcf6-pfvlz\" (UID: \"d4962250-4950-463c-92ee-b6d105f03d80\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"
Aug 04 18:00:23.191852 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.191719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:23.191852 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.191797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99cml\" (UniqueName: \"kubernetes.io/projected/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-kube-api-access-99cml\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.191852 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.191820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mph9n\" (UniqueName: \"kubernetes.io/projected/cd3e57c1-5ced-4a81-99f5-1021eb303059-kube-api-access-mph9n\") pod \"cluster-samples-operator-559cbf5745-s4r8f\" (UID: \"cd3e57c1-5ced-4a81-99f5-1021eb303059\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:23.191852 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.191838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgljk\" (UniqueName: \"kubernetes.io/projected/8c9b8163-7fde-4dea-a551-ad964becd7e0-kube-api-access-tgljk\") pod \"network-check-source-65bdbbff45-vl78f\" (UID: \"8c9b8163-7fde-4dea-a551-ad964becd7e0\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-vl78f"
Aug 04 18:00:23.192113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.191857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9kgh\" (UniqueName: \"kubernetes.io/projected/d5277a01-9455-4067-9d90-067385c2116b-kube-api-access-n9kgh\") pod \"volume-data-source-validator-5b8776669-qk8jv\" (UID: \"d5277a01-9455-4067-9d90-067385c2116b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-qk8jv"
Aug 04 18:00:23.192113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.191879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7b1bac-ba42-4f89-bc21-b06342fec05c-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xftn9\" (UID: \"0e7b1bac-ba42-4f89-bc21-b06342fec05c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"
Aug 04 18:00:23.192113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.191926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5n2h\" (UniqueName: \"kubernetes.io/projected/d4962250-4950-463c-92ee-b6d105f03d80-kube-api-access-t5n2h\") pod \"service-ca-operator-784f97bcf6-pfvlz\" (UID: \"d4962250-4950-463c-92ee-b6d105f03d80\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"
Aug 04 18:00:23.192113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e7b1bac-ba42-4f89-bc21-b06342fec05c-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xftn9\" (UID: \"0e7b1bac-ba42-4f89-bc21-b06342fec05c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"
Aug 04 18:00:23.192285 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/37559df0-a4d3-48ac-87be-21de5499def8-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:23.192285 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpdrc\" (UniqueName: \"kubernetes.io/projected/db104465-bd2b-4963-b19c-26f2b8438865-kube-api-access-vpdrc\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.192285 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s4r8f\" (UID: \"cd3e57c1-5ced-4a81-99f5-1021eb303059\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:23.192285 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcsd\" (UniqueName: \"kubernetes.io/projected/0e7b1bac-ba42-4f89-bc21-b06342fec05c-kube-api-access-2gcsd\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xftn9\" (UID: \"0e7b1bac-ba42-4f89-bc21-b06342fec05c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"
Aug 04 18:00:23.192465 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192331    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4962250-4950-463c-92ee-b6d105f03d80-config\") pod \"service-ca-operator-784f97bcf6-pfvlz\" (UID: \"d4962250-4950-463c-92ee-b6d105f03d80\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"
Aug 04 18:00:23.192465 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.192348    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 18:00:23.192465 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e8f66e5-a4db-45ee-877a-de0246618be7-tmp\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.192465 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4wl\" (UniqueName: \"kubernetes.io/projected/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-kube-api-access-7c4wl\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:23.192465 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.192404    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls podName:cd3e57c1-5ced-4a81-99f5-1021eb303059 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:23.69238538 +0000 UTC m=+49.036786515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-s4r8f" (UID: "cd3e57c1-5ced-4a81-99f5-1021eb303059") : secret "samples-operator-tls" not found
Aug 04 18:00:23.192465 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-serving-cert\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.192680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-stats-auth\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.192680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e8f66e5-a4db-45ee-877a-de0246618be7-serving-cert\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.192680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:23.192680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e8f66e5-a4db-45ee-877a-de0246618be7-trusted-ca-bundle\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.192680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-default-certificate\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.192680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e8f66e5-a4db-45ee-877a-de0246618be7-tmp\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.192680 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.192651    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:23.192680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-config\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.192987 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.192722    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls podName:2335dcc1-9d52-483e-b1c6-1e58887ec4ca nodeName:}" failed. No retries permitted until 2026-08-04 18:00:23.692706373 +0000 UTC m=+49.037107512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-wvj9z" (UID: "2335dcc1-9d52-483e-b1c6-1e58887ec4ca") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:23.192987 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e8f66e5-a4db-45ee-877a-de0246618be7-snapshots\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.192987 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-trusted-ca\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.192987 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lm6q\" (UniqueName: \"kubernetes.io/projected/7e8f66e5-a4db-45ee-877a-de0246618be7-kube-api-access-8lm6q\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.192987 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.192987 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.192943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.193260 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.193075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:23.193452 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.193427    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e8f66e5-a4db-45ee-877a-de0246618be7-snapshots\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.193635 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.193619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e8f66e5-a4db-45ee-877a-de0246618be7-trusted-ca-bundle\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.193712 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.193689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:23.196316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.196292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4962250-4950-463c-92ee-b6d105f03d80-serving-cert\") pod \"service-ca-operator-784f97bcf6-pfvlz\" (UID: \"d4962250-4950-463c-92ee-b6d105f03d80\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"
Aug 04 18:00:23.196316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.196315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e8f66e5-a4db-45ee-877a-de0246618be7-serving-cert\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.202851 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.202464    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e8f66e5-a4db-45ee-877a-de0246618be7-service-ca-bundle\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.204221 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.204192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5n2h\" (UniqueName: \"kubernetes.io/projected/d4962250-4950-463c-92ee-b6d105f03d80-kube-api-access-t5n2h\") pod \"service-ca-operator-784f97bcf6-pfvlz\" (UID: \"d4962250-4950-463c-92ee-b6d105f03d80\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"
Aug 04 18:00:23.204570 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.204549    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9kgh\" (UniqueName: \"kubernetes.io/projected/d5277a01-9455-4067-9d90-067385c2116b-kube-api-access-n9kgh\") pod \"volume-data-source-validator-5b8776669-qk8jv\" (UID: \"d5277a01-9455-4067-9d90-067385c2116b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-qk8jv"
Aug 04 18:00:23.204634 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.204591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4wl\" (UniqueName: \"kubernetes.io/projected/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-kube-api-access-7c4wl\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:23.204634 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.204610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mph9n\" (UniqueName: \"kubernetes.io/projected/cd3e57c1-5ced-4a81-99f5-1021eb303059-kube-api-access-mph9n\") pod \"cluster-samples-operator-559cbf5745-s4r8f\" (UID: \"cd3e57c1-5ced-4a81-99f5-1021eb303059\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:23.204852 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.204837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lm6q\" (UniqueName: \"kubernetes.io/projected/7e8f66e5-a4db-45ee-877a-de0246618be7-kube-api-access-8lm6q\") pod \"insights-operator-77696685b-zxjxn\" (UID: \"7e8f66e5-a4db-45ee-877a-de0246618be7\") " pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.215151 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.215122    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-qk8jv"
Aug 04 18:00:23.247567 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.247543    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"
Aug 04 18:00:23.280578 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.280538    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:23.294354 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.294322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.294524 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.294360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.294524 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.294401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:23.294524 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.294461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99cml\" (UniqueName: \"kubernetes.io/projected/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-kube-api-access-99cml\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.294524 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.294496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgljk\" (UniqueName: \"kubernetes.io/projected/8c9b8163-7fde-4dea-a551-ad964becd7e0-kube-api-access-tgljk\") pod \"network-check-source-65bdbbff45-vl78f\" (UID: \"8c9b8163-7fde-4dea-a551-ad964becd7e0\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-vl78f"
Aug 04 18:00:23.294524 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.294502    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:23.294524 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.294524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7b1bac-ba42-4f89-bc21-b06342fec05c-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xftn9\" (UID: \"0e7b1bac-ba42-4f89-bc21-b06342fec05c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"
Aug 04 18:00:23.295092 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.294554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e7b1bac-ba42-4f89-bc21-b06342fec05c-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xftn9\" (UID: \"0e7b1bac-ba42-4f89-bc21-b06342fec05c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"
Aug 04 18:00:23.295092 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.294574    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:23.794550799 +0000 UTC m=+49.138951941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:23.295092 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.294758    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:23.794743836 +0000 UTC m=+49.139144974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : secret "router-metrics-certs-default" not found
Aug 04 18:00:23.295092 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.294805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/37559df0-a4d3-48ac-87be-21de5499def8-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:23.295092 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.294841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpdrc\" (UniqueName: \"kubernetes.io/projected/db104465-bd2b-4963-b19c-26f2b8438865-kube-api-access-vpdrc\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.295092 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.294907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcsd\" (UniqueName: \"kubernetes.io/projected/0e7b1bac-ba42-4f89-bc21-b06342fec05c-kube-api-access-2gcsd\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xftn9\" (UID: \"0e7b1bac-ba42-4f89-bc21-b06342fec05c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"
Aug 04 18:00:23.295092 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.294953    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:23.295092 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.294992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-serving-cert\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.295092 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.295002    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert podName:37559df0-a4d3-48ac-87be-21de5499def8 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:23.794986987 +0000 UTC m=+49.139388123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tlnn6" (UID: "37559df0-a4d3-48ac-87be-21de5499def8") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:23.295092 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.295073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-stats-auth\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.295623 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.295117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e7b1bac-ba42-4f89-bc21-b06342fec05c-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xftn9\" (UID: \"0e7b1bac-ba42-4f89-bc21-b06342fec05c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"
Aug 04 18:00:23.295623 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.295147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-default-certificate\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.295623 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.295175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-config\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.295623 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.295204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-trusted-ca\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.296510 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.296395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-trusted-ca\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.296510 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.296469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/37559df0-a4d3-48ac-87be-21de5499def8-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:23.296744 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.296479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-config\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.300154 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.299158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-zxjxn"
Aug 04 18:00:23.300154 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.299569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-stats-auth\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.300730 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.300705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7b1bac-ba42-4f89-bc21-b06342fec05c-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xftn9\" (UID: \"0e7b1bac-ba42-4f89-bc21-b06342fec05c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"
Aug 04 18:00:23.300878 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.300857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-serving-cert\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.302808 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.302780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-default-certificate\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.308371 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.306419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99cml\" (UniqueName: \"kubernetes.io/projected/3a326eea-6141-4dca-a686-a9fa0dcdbaf6-kube-api-access-99cml\") pod \"console-operator-5ccd6b445f-xgggh\" (UID: \"3a326eea-6141-4dca-a686-a9fa0dcdbaf6\") " pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.308371 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.306906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpdrc\" (UniqueName: \"kubernetes.io/projected/db104465-bd2b-4963-b19c-26f2b8438865-kube-api-access-vpdrc\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.308371 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.307686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgljk\" (UniqueName: \"kubernetes.io/projected/8c9b8163-7fde-4dea-a551-ad964becd7e0-kube-api-access-tgljk\") pod \"network-check-source-65bdbbff45-vl78f\" (UID: \"8c9b8163-7fde-4dea-a551-ad964becd7e0\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-vl78f"
Aug 04 18:00:23.308371 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.307812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcsd\" (UniqueName: \"kubernetes.io/projected/0e7b1bac-ba42-4f89-bc21-b06342fec05c-kube-api-access-2gcsd\") pod \"kube-storage-version-migrator-operator-7878f4b88c-xftn9\" (UID: \"0e7b1bac-ba42-4f89-bc21-b06342fec05c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"
Aug 04 18:00:23.318512 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.318422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:23.360690 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.360641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"
Aug 04 18:00:23.415123 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.414134    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-qk8jv"]
Aug 04 18:00:23.415123 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.414274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-vl78f"
Aug 04 18:00:23.428943 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:23.428909    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5277a01_9455_4067_9d90_067385c2116b.slice/crio-caf2be8997b00651dd30f8b84438f3bc7ffb4d89de689f48befb77fda6a6ce32 WatchSource:0}: Error finding container caf2be8997b00651dd30f8b84438f3bc7ffb4d89de689f48befb77fda6a6ce32: Status 404 returned error can't find the container with id caf2be8997b00651dd30f8b84438f3bc7ffb4d89de689f48befb77fda6a6ce32
Aug 04 18:00:23.470070 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.469312    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz"]
Aug 04 18:00:23.474477 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:23.474258    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4962250_4950_463c_92ee_b6d105f03d80.slice/crio-5bbfdca8ee3d53dd84afd44122572b6ff3f1bde1bdff46eab4db1a7ec48134c8 WatchSource:0}: Error finding container 5bbfdca8ee3d53dd84afd44122572b6ff3f1bde1bdff46eab4db1a7ec48134c8: Status 404 returned error can't find the container with id 5bbfdca8ee3d53dd84afd44122572b6ff3f1bde1bdff46eab4db1a7ec48134c8
Aug 04 18:00:23.515233 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.513658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz" event={"ID":"d4962250-4950-463c-92ee-b6d105f03d80","Type":"ContainerStarted","Data":"5bbfdca8ee3d53dd84afd44122572b6ff3f1bde1bdff46eab4db1a7ec48134c8"}
Aug 04 18:00:23.515233 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.515115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-qk8jv" event={"ID":"d5277a01-9455-4067-9d90-067385c2116b","Type":"ContainerStarted","Data":"caf2be8997b00651dd30f8b84438f3bc7ffb4d89de689f48befb77fda6a6ce32"}
Aug 04 18:00:23.530137 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.528661    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-zxjxn"]
Aug 04 18:00:23.534823 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:23.534781    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8f66e5_a4db_45ee_877a_de0246618be7.slice/crio-19c58da60d4d1dbff05b2e1b908f3d50efd3cddad94558765f2004fdedb8363b WatchSource:0}: Error finding container 19c58da60d4d1dbff05b2e1b908f3d50efd3cddad94558765f2004fdedb8363b: Status 404 returned error can't find the container with id 19c58da60d4d1dbff05b2e1b908f3d50efd3cddad94558765f2004fdedb8363b
Aug 04 18:00:23.560865 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.560784    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-xgggh"]
Aug 04 18:00:23.564715 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:23.564676    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a326eea_6141_4dca_a686_a9fa0dcdbaf6.slice/crio-855fb080f1177a9184034e9e744bbc13af0ea24016ebc0b5b66f8d3559481651 WatchSource:0}: Error finding container 855fb080f1177a9184034e9e744bbc13af0ea24016ebc0b5b66f8d3559481651: Status 404 returned error can't find the container with id 855fb080f1177a9184034e9e744bbc13af0ea24016ebc0b5b66f8d3559481651
Aug 04 18:00:23.600625 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.600596    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9"]
Aug 04 18:00:23.603894 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:23.603865    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7b1bac_ba42_4f89_bc21_b06342fec05c.slice/crio-8dba33df03910efe37d3fb39df75efa723d2ce760631502c47bab002ef41b6cc WatchSource:0}: Error finding container 8dba33df03910efe37d3fb39df75efa723d2ce760631502c47bab002ef41b6cc: Status 404 returned error can't find the container with id 8dba33df03910efe37d3fb39df75efa723d2ce760631502c47bab002ef41b6cc
Aug 04 18:00:23.615061 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.615016    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-vl78f"]
Aug 04 18:00:23.618703 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:23.618676    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9b8163_7fde_4dea_a551_ad964becd7e0.slice/crio-0cb9f54793dacba0a1394acbc584d46b9c6b12e6ca77fed4a285ade9b2a81aed WatchSource:0}: Error finding container 0cb9f54793dacba0a1394acbc584d46b9c6b12e6ca77fed4a285ade9b2a81aed: Status 404 returned error can't find the container with id 0cb9f54793dacba0a1394acbc584d46b9c6b12e6ca77fed4a285ade9b2a81aed
Aug 04 18:00:23.699502 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.699414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s4r8f\" (UID: \"cd3e57c1-5ced-4a81-99f5-1021eb303059\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:23.699502 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.699484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:23.699681 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.699571    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:23.699681 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.699572    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 18:00:23.699681 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.699624    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls podName:2335dcc1-9d52-483e-b1c6-1e58887ec4ca nodeName:}" failed. No retries permitted until 2026-08-04 18:00:24.699611269 +0000 UTC m=+50.044012404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-wvj9z" (UID: "2335dcc1-9d52-483e-b1c6-1e58887ec4ca") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:23.699681 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.699638    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls podName:cd3e57c1-5ced-4a81-99f5-1021eb303059 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:24.699631993 +0000 UTC m=+50.044033127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-s4r8f" (UID: "cd3e57c1-5ced-4a81-99f5-1021eb303059") : secret "samples-operator-tls" not found
Aug 04 18:00:23.800676 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.800639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.800676 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.800674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:23.800874 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.800780    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:23.800874 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.800817    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:24.800798634 +0000 UTC m=+50.145199772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:23.800874 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.800846    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:24.800836337 +0000 UTC m=+50.145237478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : secret "router-metrics-certs-default" not found
Aug 04 18:00:23.800874 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:23.800812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:23.801075 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.800889    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:23.801075 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:23.801032    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert podName:37559df0-a4d3-48ac-87be-21de5499def8 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:24.800999001 +0000 UTC m=+50.145400140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tlnn6" (UID: "37559df0-a4d3-48ac-87be-21de5499def8") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:24.519595 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:24.519560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-vl78f" event={"ID":"8c9b8163-7fde-4dea-a551-ad964becd7e0","Type":"ContainerStarted","Data":"0cb9f54793dacba0a1394acbc584d46b9c6b12e6ca77fed4a285ade9b2a81aed"}
Aug 04 18:00:24.521120 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:24.521015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-zxjxn" event={"ID":"7e8f66e5-a4db-45ee-877a-de0246618be7","Type":"ContainerStarted","Data":"19c58da60d4d1dbff05b2e1b908f3d50efd3cddad94558765f2004fdedb8363b"}
Aug 04 18:00:24.522403 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:24.522375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9" event={"ID":"0e7b1bac-ba42-4f89-bc21-b06342fec05c","Type":"ContainerStarted","Data":"8dba33df03910efe37d3fb39df75efa723d2ce760631502c47bab002ef41b6cc"}
Aug 04 18:00:24.523727 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:24.523701    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh" event={"ID":"3a326eea-6141-4dca-a686-a9fa0dcdbaf6","Type":"ContainerStarted","Data":"855fb080f1177a9184034e9e744bbc13af0ea24016ebc0b5b66f8d3559481651"}
Aug 04 18:00:24.709571 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:24.709529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:24.709786 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:24.709694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s4r8f\" (UID: \"cd3e57c1-5ced-4a81-99f5-1021eb303059\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:24.709871 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:24.709853    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 18:00:24.709934 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:24.709926    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls podName:cd3e57c1-5ced-4a81-99f5-1021eb303059 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:26.709905645 +0000 UTC m=+52.054306786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-s4r8f" (UID: "cd3e57c1-5ced-4a81-99f5-1021eb303059") : secret "samples-operator-tls" not found
Aug 04 18:00:24.712393 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:24.712362    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:24.712531 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:24.712458    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls podName:2335dcc1-9d52-483e-b1c6-1e58887ec4ca nodeName:}" failed. No retries permitted until 2026-08-04 18:00:26.71243498 +0000 UTC m=+52.056836118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-wvj9z" (UID: "2335dcc1-9d52-483e-b1c6-1e58887ec4ca") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:24.811020 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:24.810329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:24.811020 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:24.810379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:24.811020 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:24.810417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:24.811020 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:24.810524    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:24.811020 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:24.810571    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert podName:37559df0-a4d3-48ac-87be-21de5499def8 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:26.810558494 +0000 UTC m=+52.154959628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tlnn6" (UID: "37559df0-a4d3-48ac-87be-21de5499def8") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:24.811620 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:24.811529    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:24.811620 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:24.811596    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:26.811578404 +0000 UTC m=+52.155979543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : secret "router-metrics-certs-default" not found
Aug 04 18:00:24.811864 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:24.811838    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:26.811823189 +0000 UTC m=+52.156224324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:25.396258 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:25.396166    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:25.502527 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:25.502371    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:25.529853 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:25.529375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-qk8jv" event={"ID":"d5277a01-9455-4067-9d90-067385c2116b","Type":"ContainerStarted","Data":"dc1ee53203a4ec6668c0a6ca0919b8e5664f82da644c1c9e9d74168ff6b72d1f"}
Aug 04 18:00:25.546248 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:25.546152    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-qk8jv" podStartSLOduration=1.817350463 podStartE2EDuration="3.546128034s" totalImagesPullingTime="1.728777571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:25.544498687 +0000 UTC m=+50.888899844" watchObservedRunningTime="2026-08-04 18:00:25.546128034 +0000 UTC m=+50.890529193"
Aug 04 18:00:25.577151 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:25.577112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:25.652737 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:25.652696    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:26.412804 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:26.412756    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:26.475844 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:26.475809    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:26.576766 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:26.576718    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:26.731784 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:26.731691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:26.731941 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:26.731845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s4r8f\" (UID: \"cd3e57c1-5ced-4a81-99f5-1021eb303059\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:26.731941 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.731859    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:26.731941 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.731928    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls podName:2335dcc1-9d52-483e-b1c6-1e58887ec4ca nodeName:}" failed. No retries permitted until 2026-08-04 18:00:30.731907795 +0000 UTC m=+56.076308950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-wvj9z" (UID: "2335dcc1-9d52-483e-b1c6-1e58887ec4ca") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:26.732102 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.731984    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 18:00:26.732102 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.732059    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls podName:cd3e57c1-5ced-4a81-99f5-1021eb303059 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:30.732028021 +0000 UTC m=+56.076429156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-s4r8f" (UID: "cd3e57c1-5ced-4a81-99f5-1021eb303059") : secret "samples-operator-tls" not found
Aug 04 18:00:26.832563 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:26.832522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:26.832755 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:26.832573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:26.832755 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:26.832614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:26.832755 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.832669    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:30.832648179 +0000 UTC m=+56.177049317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:26.832755 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.832726    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:26.832755 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.832731    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:26.832977 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.832777    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert podName:37559df0-a4d3-48ac-87be-21de5499def8 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:30.832761418 +0000 UTC m=+56.177162553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tlnn6" (UID: "37559df0-a4d3-48ac-87be-21de5499def8") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:26.832977 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.832795    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:30.832784902 +0000 UTC m=+56.177186043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : secret "router-metrics-certs-default" not found
Aug 04 18:00:26.933614 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:26.933579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:26.933821 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:26.933644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:26.933821 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.933754    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:26.933905 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.933828    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls podName:22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:42.933801417 +0000 UTC m=+68.278202552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls") pod "dns-default-5x45c" (UID: "22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:26.933905 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.933757    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:26.933905 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.933848    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b8c87c4-9gpcq: secret "image-registry-tls" not found
Aug 04 18:00:26.933905 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:26.933878    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls podName:ecebf12c-a9cc-44dd-bee5-d210c3209aa9 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:42.933871861 +0000 UTC m=+68.278272995 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls") pod "image-registry-565b8c87c4-9gpcq" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9") : secret "image-registry-tls" not found
Aug 04 18:00:27.431860 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:27.431820    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:30.532891 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.532850    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f767j"]
Aug 04 18:00:30.536360 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.536333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.538762 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.538733    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9fk8\""
Aug 04 18:00:30.538915 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.538861    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 04 18:00:30.539560 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.539533    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 04 18:00:30.545663 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.545634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz" event={"ID":"d4962250-4950-463c-92ee-b6d105f03d80","Type":"ContainerStarted","Data":"4929e8de6e21c4a67f00d98f622361a07f6c8a624ca5a51bd0e8c1ff9c163b93"}
Aug 04 18:00:30.547336 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.547301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:30.549023 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.548987    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f767j"]
Aug 04 18:00:30.549793 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.549767    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9" event={"ID":"0e7b1bac-ba42-4f89-bc21-b06342fec05c","Type":"ContainerStarted","Data":"fe48117984df7ec50ecbbf438da53b98a1639174bf947ce232ca3f0a462359e0"}
Aug 04 18:00:30.551996 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.551976    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/0.log"
Aug 04 18:00:30.552117 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.552021    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a326eea-6141-4dca-a686-a9fa0dcdbaf6" containerID="049c918b6db50395e2786557aa87771a04eefdf763614c85036791246dcc27aa" exitCode=255
Aug 04 18:00:30.552117 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.552073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh" event={"ID":"3a326eea-6141-4dca-a686-a9fa0dcdbaf6","Type":"ContainerDied","Data":"049c918b6db50395e2786557aa87771a04eefdf763614c85036791246dcc27aa"}
Aug 04 18:00:30.552375 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.552361    2581 scope.go:117] "RemoveContainer" containerID="049c918b6db50395e2786557aa87771a04eefdf763614c85036791246dcc27aa"
Aug 04 18:00:30.554876 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.554829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-vl78f" event={"ID":"8c9b8163-7fde-4dea-a551-ad964becd7e0","Type":"ContainerStarted","Data":"4877ab54274720705688802d14100625721a649d43217e139b9fe16529067f56"}
Aug 04 18:00:30.557174 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.557150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-zxjxn" event={"ID":"7e8f66e5-a4db-45ee-877a-de0246618be7","Type":"ContainerStarted","Data":"2ce3bc8192d881ecfb16d7e107bfaaffa14d93f589d40ccd6a608a8a252731a5"}
Aug 04 18:00:30.569288 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.567770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87611d06-12e3-4b27-8ce5-a156441f8b9e-crio-socket\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.569288 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.567835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.569288 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.567944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87611d06-12e3-4b27-8ce5-a156441f8b9e-data-volume\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.569288 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.568083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfjx6\" (UniqueName: \"kubernetes.io/projected/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-api-access-pfjx6\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.569288 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.568236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.582867 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.582566    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9" podStartSLOduration=2.478023008 podStartE2EDuration="8.582544008s" totalImagesPullingTime="6.104521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:30.581466673 +0000 UTC m=+55.925867830" watchObservedRunningTime="2026-08-04 18:00:30.582544008 +0000 UTC m=+55.926945144"
Aug 04 18:00:30.622008 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.621976    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:30.629317 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.629259    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-vl78f" podStartSLOduration=1.528425296 podStartE2EDuration="7.629238197s" totalImagesPullingTime="6.100812901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:30.629098135 +0000 UTC m=+55.973499292" watchObservedRunningTime="2026-08-04 18:00:30.629238197 +0000 UTC m=+55.973639355"
Aug 04 18:00:30.653257 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.653015    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz" podStartSLOduration=2.41570274 podStartE2EDuration="8.652993335s" totalImagesPullingTime="6.237290595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:30.652306561 +0000 UTC m=+55.996707713" watchObservedRunningTime="2026-08-04 18:00:30.652993335 +0000 UTC m=+55.997394494"
Aug 04 18:00:30.668700 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.668664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.669020 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.669004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87611d06-12e3-4b27-8ce5-a156441f8b9e-data-volume\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.669239 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.669224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfjx6\" (UniqueName: \"kubernetes.io/projected/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-api-access-pfjx6\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.669422 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.669410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.669547 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.669534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87611d06-12e3-4b27-8ce5-a156441f8b9e-crio-socket\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.669743 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.669730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87611d06-12e3-4b27-8ce5-a156441f8b9e-crio-socket\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.669922 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.669910    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:30.670131 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.670037    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls podName:87611d06-12e3-4b27-8ce5-a156441f8b9e nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.170016349 +0000 UTC m=+56.514417497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f767j" (UID: "87611d06-12e3-4b27-8ce5-a156441f8b9e") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:30.671322 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.671304    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:30.671492 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.671481    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm podName:87611d06-12e3-4b27-8ce5-a156441f8b9e nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.171461851 +0000 UTC m=+56.515862993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f767j" (UID: "87611d06-12e3-4b27-8ce5-a156441f8b9e") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:30.671874 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.671857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87611d06-12e3-4b27-8ce5-a156441f8b9e-data-volume\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.678670 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.678611    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-zxjxn" podStartSLOduration=2.505465907 podStartE2EDuration="8.678591076s" totalImagesPullingTime="6.173125169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:30.674585599 +0000 UTC m=+56.018986752" watchObservedRunningTime="2026-08-04 18:00:30.678591076 +0000 UTC m=+56.022992235"
Aug 04 18:00:30.690412 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.688628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfjx6\" (UniqueName: \"kubernetes.io/projected/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-api-access-pfjx6\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:30.771017 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.770975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s4r8f\" (UID: \"cd3e57c1-5ced-4a81-99f5-1021eb303059\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:30.771213 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.771087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:30.771213 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.771166    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 18:00:30.771213 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.771204    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:30.771406 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.771254    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls podName:cd3e57c1-5ced-4a81-99f5-1021eb303059 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.77123253 +0000 UTC m=+64.115633680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-s4r8f" (UID: "cd3e57c1-5ced-4a81-99f5-1021eb303059") : secret "samples-operator-tls" not found
Aug 04 18:00:30.771406 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.771290    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls podName:2335dcc1-9d52-483e-b1c6-1e58887ec4ca nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.771278477 +0000 UTC m=+64.115679615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-wvj9z" (UID: "2335dcc1-9d52-483e-b1c6-1e58887ec4ca") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:30.872657 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.872559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:30.872657 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.872607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:30.872886 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:30.872659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:30.872886 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.872762    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.872737343 +0000 UTC m=+64.217138482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:30.872886 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.872805    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:30.872886 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.872812    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:30.873068 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.872889    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.872877383 +0000 UTC m=+64.217278524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : secret "router-metrics-certs-default" not found
Aug 04 18:00:30.873068 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:30.872940    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert podName:37559df0-a4d3-48ac-87be-21de5499def8 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.872921441 +0000 UTC m=+64.217322588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tlnn6" (UID: "37559df0-a4d3-48ac-87be-21de5499def8") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:31.071692 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.071660    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:31.169609 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.169519    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:31.175412 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.175385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:31.175558 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:31.175540    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:31.175625 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.175561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:31.175625 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:31.175605    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls podName:87611d06-12e3-4b27-8ce5-a156441f8b9e nodeName:}" failed. No retries permitted until 2026-08-04 18:00:32.175588876 +0000 UTC m=+57.519990010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f767j" (UID: "87611d06-12e3-4b27-8ce5-a156441f8b9e") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:31.175732 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:31.175637    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:31.175732 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:31.175696    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm podName:87611d06-12e3-4b27-8ce5-a156441f8b9e nodeName:}" failed. No retries permitted until 2026-08-04 18:00:32.175679028 +0000 UTC m=+57.520080177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f767j" (UID: "87611d06-12e3-4b27-8ce5-a156441f8b9e") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:31.279720 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.279684    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:31.372461 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.372427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:31.460217 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.460091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:31.562278 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.562248    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:00:31.562671 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.562653    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/0.log"
Aug 04 18:00:31.562727 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.562685    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a326eea-6141-4dca-a686-a9fa0dcdbaf6" containerID="d979426df2e2d85d66711989be8500e92c4981e4215be5eb933e75c6e3c3b51d" exitCode=255
Aug 04 18:00:31.562818 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.562796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh" event={"ID":"3a326eea-6141-4dca-a686-a9fa0dcdbaf6","Type":"ContainerDied","Data":"d979426df2e2d85d66711989be8500e92c4981e4215be5eb933e75c6e3c3b51d"}
Aug 04 18:00:31.562872 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.562850    2581 scope.go:117] "RemoveContainer" containerID="049c918b6db50395e2786557aa87771a04eefdf763614c85036791246dcc27aa"
Aug 04 18:00:31.563626 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.563210    2581 scope.go:117] "RemoveContainer" containerID="d979426df2e2d85d66711989be8500e92c4981e4215be5eb933e75c6e3c3b51d"
Aug 04 18:00:31.563626 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:31.563412    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-xgggh_openshift-console-operator(3a326eea-6141-4dca-a686-a9fa0dcdbaf6)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh" podUID="3a326eea-6141-4dca-a686-a9fa0dcdbaf6"
Aug 04 18:00:31.774630 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.774599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:31.875291 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:31.875262    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:32.109914 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:32.109839    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nnr9j_883287f9-439b-406f-92be-57c88b121f72/dns-node-resolver/0.log"
Aug 04 18:00:32.186290 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:32.186256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:32.186497 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:32.186354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:32.186497 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:32.186389    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:32.186497 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:32.186470    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm podName:87611d06-12e3-4b27-8ce5-a156441f8b9e nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.186449317 +0000 UTC m=+59.530850462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f767j" (UID: "87611d06-12e3-4b27-8ce5-a156441f8b9e") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:32.186626 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:32.186503    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:32.186626 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:32.186542    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls podName:87611d06-12e3-4b27-8ce5-a156441f8b9e nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.186533418 +0000 UTC m=+59.530934554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f767j" (UID: "87611d06-12e3-4b27-8ce5-a156441f8b9e") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:32.566966 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:32.566939    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:00:32.567452 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:32.567383    2581 scope.go:117] "RemoveContainer" containerID="d979426df2e2d85d66711989be8500e92c4981e4215be5eb933e75c6e3c3b51d"
Aug 04 18:00:32.567602 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:32.567579    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-xgggh_openshift-console-operator(3a326eea-6141-4dca-a686-a9fa0dcdbaf6)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh" podUID="3a326eea-6141-4dca-a686-a9fa0dcdbaf6"
Aug 04 18:00:33.309792 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:33.309765    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gwkqx_6516e99e-3bc1-46a1-ad86-2f72e109c596/node-ca/0.log"
Aug 04 18:00:33.318932 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:33.318901    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:33.319103 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:33.318999    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:33.474141 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:33.474102    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:33.541428 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:33.541393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:33.571217 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:33.571149    2581 scope.go:117] "RemoveContainer" containerID="d979426df2e2d85d66711989be8500e92c4981e4215be5eb933e75c6e3c3b51d"
Aug 04 18:00:33.571530 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:33.571319    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-xgggh_openshift-console-operator(3a326eea-6141-4dca-a686-a9fa0dcdbaf6)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh" podUID="3a326eea-6141-4dca-a686-a9fa0dcdbaf6"
Aug 04 18:00:33.641746 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:33.641712    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:33.712006 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:33.711977    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8brzb_d80c332a-acaa-4ecb-b197-e69f0c4116d5/serve-healthcheck-canary/0.log"
Aug 04 18:00:34.205813 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:34.205764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:34.205993 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:34.205864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:34.205993 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:34.205905    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:34.206130 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:34.205993    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm podName:87611d06-12e3-4b27-8ce5-a156441f8b9e nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.205969257 +0000 UTC m=+63.550370415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f767j" (UID: "87611d06-12e3-4b27-8ce5-a156441f8b9e") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:34.206130 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:34.206011    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:34.206130 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:34.206093    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls podName:87611d06-12e3-4b27-8ce5-a156441f8b9e nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.206080922 +0000 UTC m=+63.550482060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f767j" (UID: "87611d06-12e3-4b27-8ce5-a156441f8b9e") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:34.457659 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:34.457559    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:34.520725 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:34.520688    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:34.574572 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:34.574546    2581 scope.go:117] "RemoveContainer" containerID="d979426df2e2d85d66711989be8500e92c4981e4215be5eb933e75c6e3c3b51d"
Aug 04 18:00:34.574979 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:34.574699    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-xgggh_openshift-console-operator(3a326eea-6141-4dca-a686-a9fa0dcdbaf6)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh" podUID="3a326eea-6141-4dca-a686-a9fa0dcdbaf6"
Aug 04 18:00:34.995494 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:34.995461    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:36.466944 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:36.466916    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2wfn"
Aug 04 18:00:37.932333 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:37.932292    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:38.033278 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.033237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:38.129752 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.129711    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:38.245992 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.245868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:38.245992 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.245984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:38.246250 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:38.246006    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:38.246250 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:38.246116    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm podName:87611d06-12e3-4b27-8ce5-a156441f8b9e nodeName:}" failed. No retries permitted until 2026-08-04 18:00:46.246098325 +0000 UTC m=+71.590499463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f767j" (UID: "87611d06-12e3-4b27-8ce5-a156441f8b9e") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:38.246250 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:38.246146    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:38.246250 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:38.246223    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls podName:87611d06-12e3-4b27-8ce5-a156441f8b9e nodeName:}" failed. No retries permitted until 2026-08-04 18:00:46.246200596 +0000 UTC m=+71.590601731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f767j" (UID: "87611d06-12e3-4b27-8ce5-a156441f8b9e") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:38.852346 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.852284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s4r8f\" (UID: \"cd3e57c1-5ced-4a81-99f5-1021eb303059\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:38.852568 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.852412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:38.852568 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:38.852526    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:38.852682 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:38.852614    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls podName:2335dcc1-9d52-483e-b1c6-1e58887ec4ca nodeName:}" failed. No retries permitted until 2026-08-04 18:00:54.852592813 +0000 UTC m=+80.196993947 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-wvj9z" (UID: "2335dcc1-9d52-483e-b1c6-1e58887ec4ca") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:38.854943 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.854919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd3e57c1-5ced-4a81-99f5-1021eb303059-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s4r8f\" (UID: \"cd3e57c1-5ced-4a81-99f5-1021eb303059\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:38.889711 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.889681    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pdpmd\""
Aug 04 18:00:38.898704 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.898671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"
Aug 04 18:00:38.952918 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.952869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:38.953402 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.953030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:38.953402 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:38.953060    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:38.953402 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:38.953132    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:38.953402 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:38.953138    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert podName:37559df0-a4d3-48ac-87be-21de5499def8 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:54.953116857 +0000 UTC m=+80.297518013 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-tlnn6" (UID: "37559df0-a4d3-48ac-87be-21de5499def8") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:38.953402 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:38.953187    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:54.953175215 +0000 UTC m=+80.297576350 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : secret "router-metrics-certs-default" not found
Aug 04 18:00:38.953402 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:38.953242    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle podName:db104465-bd2b-4963-b19c-26f2b8438865 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:54.953233998 +0000 UTC m=+80.297635136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle") pod "router-default-785c67bddb-zlhb4" (UID: "db104465-bd2b-4963-b19c-26f2b8438865") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:38.953402 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:38.953069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:39.031903 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:39.031872    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f"]
Aug 04 18:00:39.151641 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:39.151520    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:39.251988 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:39.251954    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:39.592286 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:39.592250    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f" event={"ID":"cd3e57c1-5ced-4a81-99f5-1021eb303059","Type":"ContainerStarted","Data":"0222546fed60779b291b6c116833a2fe5bcda1a63b3345acfdb7059112abd1d1"}
Aug 04 18:00:39.635957 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:39.635911    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:39.828689 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:39.828649    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:40.225591 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:40.225546    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:40.448767 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:40.448729    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:40.840955 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:40.840849    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:40.974094 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:40.974060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:00:40.974272 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:40.974161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 18:00:40.976737 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:40.976714    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:00:40.976842 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:40.976768    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:00:40.984848 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:40.984817    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 04 18:00:40.984994 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:40.984891    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs podName:56e86da9-f3b0-4819-872b-8f4944af85e8 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:44.984873875 +0000 UTC m=+130.329275010 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs") pod "network-metrics-daemon-gdpj9" (UID: "56e86da9-f3b0-4819-872b-8f4944af85e8") : secret "metrics-daemon-secret" not found
Aug 04 18:00:40.987302 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:40.987272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8626a82d-8bbc-4e96-bd2d-bf7a1ec40786-original-pull-secret\") pod \"global-pull-secret-syncer-ksxcz\" (UID: \"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786\") " pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 18:00:41.175952 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.175927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75kw\" (UniqueName: \"kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw\") pod \"network-check-target-gfgc5\" (UID: \"fdbe2137-0bf9-4a65-a12b-684dc3da0657\") " pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:41.178822 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.178793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75kw\" (UniqueName: \"kubernetes.io/projected/fdbe2137-0bf9-4a65-a12b-684dc3da0657-kube-api-access-f75kw\") pod \"network-check-target-gfgc5\" (UID: \"fdbe2137-0bf9-4a65-a12b-684dc3da0657\") " pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:41.232483 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.232445    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxcz"
Aug 04 18:00:41.243078 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.243032    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h457s\""
Aug 04 18:00:41.250897 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.250863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:41.373926 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.373894    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ksxcz"]
Aug 04 18:00:41.377481 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:41.377453    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8626a82d_8bbc_4e96_bd2d_bf7a1ec40786.slice/crio-b8c01e2f954fc6e6fa01bda7bb5cdb7b3a6ce8547f2a366c48726165fd4fb8f0 WatchSource:0}: Error finding container b8c01e2f954fc6e6fa01bda7bb5cdb7b3a6ce8547f2a366c48726165fd4fb8f0: Status 404 returned error can't find the container with id b8c01e2f954fc6e6fa01bda7bb5cdb7b3a6ce8547f2a366c48726165fd4fb8f0
Aug 04 18:00:41.425822 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.425798    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gfgc5"]
Aug 04 18:00:41.426590 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.426563    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:41.427870 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:41.427848    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbe2137_0bf9_4a65_a12b_684dc3da0657.slice/crio-c12972c68da420f08d30be055c8e4abd5d420198254198fb912e57aa9b934f20 WatchSource:0}: Error finding container c12972c68da420f08d30be055c8e4abd5d420198254198fb912e57aa9b934f20: Status 404 returned error can't find the container with id c12972c68da420f08d30be055c8e4abd5d420198254198fb912e57aa9b934f20
Aug 04 18:00:41.601746 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.601715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f" event={"ID":"cd3e57c1-5ced-4a81-99f5-1021eb303059","Type":"ContainerStarted","Data":"36f894769f3896be3becab3d546ef611b0fc2bb2df6bfde1b0b8e7fdd8e4c9a6"}
Aug 04 18:00:41.601925 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.601814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f" event={"ID":"cd3e57c1-5ced-4a81-99f5-1021eb303059","Type":"ContainerStarted","Data":"62c1c5232cf484a4404b89099c4b870933abd0adb80b9bd134360b7fb2cf36b3"}
Aug 04 18:00:41.603331 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.603304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gfgc5" event={"ID":"fdbe2137-0bf9-4a65-a12b-684dc3da0657","Type":"ContainerStarted","Data":"e2fcbca51956f10ca88ac644b34c59505048be9f7b770572ff2386c4c4ad76e8"}
Aug 04 18:00:41.603444 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.603343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gfgc5" event={"ID":"fdbe2137-0bf9-4a65-a12b-684dc3da0657","Type":"ContainerStarted","Data":"c12972c68da420f08d30be055c8e4abd5d420198254198fb912e57aa9b934f20"}
Aug 04 18:00:41.603444 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.603392    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:00:41.604480 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.604461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ksxcz" event={"ID":"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786","Type":"ContainerStarted","Data":"b8c01e2f954fc6e6fa01bda7bb5cdb7b3a6ce8547f2a366c48726165fd4fb8f0"}
Aug 04 18:00:41.622111 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.621996    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s4r8f" podStartSLOduration=17.686036077 podStartE2EDuration="19.621979752s" totalImagesPullingTime="1.935943675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:41.620617445 +0000 UTC m=+66.965018602" watchObservedRunningTime="2026-08-04 18:00:41.621979752 +0000 UTC m=+66.966380908"
Aug 04 18:00:41.628689 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.628659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:41.637246 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:41.637201    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gfgc5" podStartSLOduration=65.637184821 podStartE2EDuration="1m5.637184821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:41.636379118 +0000 UTC m=+66.980780275" watchObservedRunningTime="2026-08-04 18:00:41.637184821 +0000 UTC m=+66.981585979"
Aug 04 18:00:42.011903 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:42.011816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:42.325510 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:42.325470    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:42.427450 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:42.427418    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:42.629338 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:42.629253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:42.992823 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:42.992725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:42.992823 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:42.992804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:42.995559 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:42.995531    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7-metrics-tls\") pod \"dns-default-5x45c\" (UID: \"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7\") " pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:42.996398 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:42.996364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls\") pod \"image-registry-565b8c87c4-9gpcq\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") " pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:43.141745 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:43.141706    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cp5kp\""
Aug 04 18:00:43.149742 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:43.149717    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:43.162262 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:43.162239    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g9rrt\""
Aug 04 18:00:43.170303 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:43.170271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:43.315269 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:43.315232    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-565b8c87c4-9gpcq"]
Aug 04 18:00:43.319281 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:43.319242    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecebf12c_a9cc_44dd_bee5_d210c3209aa9.slice/crio-52bfbae8df8b6dfe09e4d22697027592cc136fd782c8f626b85feed48af2d8db WatchSource:0}: Error finding container 52bfbae8df8b6dfe09e4d22697027592cc136fd782c8f626b85feed48af2d8db: Status 404 returned error can't find the container with id 52bfbae8df8b6dfe09e4d22697027592cc136fd782c8f626b85feed48af2d8db
Aug 04 18:00:43.332230 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:43.332196    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5x45c"]
Aug 04 18:00:43.335630 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:43.335597    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f5bf10_2cd9_45c2_b2f0_7f29b134b6e7.slice/crio-39afdd6630e9cf0bc402f78fdefe035be1d5c43d6a299206fcce087799e276f4 WatchSource:0}: Error finding container 39afdd6630e9cf0bc402f78fdefe035be1d5c43d6a299206fcce087799e276f4: Status 404 returned error can't find the container with id 39afdd6630e9cf0bc402f78fdefe035be1d5c43d6a299206fcce087799e276f4
Aug 04 18:00:43.614354 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:43.614314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq" event={"ID":"ecebf12c-a9cc-44dd-bee5-d210c3209aa9","Type":"ContainerStarted","Data":"3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb"}
Aug 04 18:00:43.614528 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:43.614377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq" event={"ID":"ecebf12c-a9cc-44dd-bee5-d210c3209aa9","Type":"ContainerStarted","Data":"52bfbae8df8b6dfe09e4d22697027592cc136fd782c8f626b85feed48af2d8db"}
Aug 04 18:00:43.614528 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:43.614408    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:00:43.616110 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:43.616081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5x45c" event={"ID":"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7","Type":"ContainerStarted","Data":"39afdd6630e9cf0bc402f78fdefe035be1d5c43d6a299206fcce087799e276f4"}
Aug 04 18:00:43.637852 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:43.637786    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq" podStartSLOduration=67.637765119 podStartE2EDuration="1m7.637765119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:43.636502126 +0000 UTC m=+68.980903286" watchObservedRunningTime="2026-08-04 18:00:43.637765119 +0000 UTC m=+68.982166277"
Aug 04 18:00:44.166450 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:44.166409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:44.267332 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:44.267295    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:44.424058 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:44.423954    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:44.948352 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:44.948313    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:45.246532 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:45.246447    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:46.210012 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.209996    2581 scope.go:117] "RemoveContainer" containerID="d979426df2e2d85d66711989be8500e92c4981e4215be5eb933e75c6e3c3b51d"
Aug 04 18:00:46.324370 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.324339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:46.324565 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:46.324540    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:46.324643 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:00:46.324627    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls podName:87611d06-12e3-4b27-8ce5-a156441f8b9e nodeName:}" failed. No retries permitted until 2026-08-04 18:01:02.324605562 +0000 UTC m=+87.669006697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f767j" (UID: "87611d06-12e3-4b27-8ce5-a156441f8b9e") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:46.325035 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.325015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:46.325539 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.325507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87611d06-12e3-4b27-8ce5-a156441f8b9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:00:46.629236 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.629202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ksxcz" event={"ID":"8626a82d-8bbc-4e96-bd2d-bf7a1ec40786","Type":"ContainerStarted","Data":"5c6703030598805c54b46308821677d6002400db515fc2772f5fd589ffa573d0"}
Aug 04 18:00:46.631000 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.630974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5x45c" event={"ID":"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7","Type":"ContainerStarted","Data":"ca5d3bdb1a1af9135dc34bbef550f7d963ed25201e2b7207ac3a04c5e993d9f3"}
Aug 04 18:00:46.631160 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.631031    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:46.631160 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.631080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5x45c" event={"ID":"22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7","Type":"ContainerStarted","Data":"1f838cb5817cb12d48ad7a018ed40745775a0bb04e1b8e86dd7921c36c5853cf"}
Aug 04 18:00:46.632830 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.632813    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:00:46.632896 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.632853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh" event={"ID":"3a326eea-6141-4dca-a686-a9fa0dcdbaf6","Type":"ContainerStarted","Data":"3878cb98ca69388dd73ce5da60fbb87f341f842b1e611d88b65b7ed1678de18c"}
Aug 04 18:00:46.633100 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.633070    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:46.646011 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.645962    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ksxcz" podStartSLOduration=65.896779316 podStartE2EDuration="1m10.645946723s" totalImagesPullingTime="4.749167407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:46.645415265 +0000 UTC m=+71.989816424" watchObservedRunningTime="2026-08-04 18:00:46.645946723 +0000 UTC m=+71.990347880"
Aug 04 18:00:46.663818 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.663764    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5x45c" podStartSLOduration=33.877817725 podStartE2EDuration="36.663749936s" totalImagesPullingTime="2.785932211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:46.662245887 +0000 UTC m=+72.006647022" watchObservedRunningTime="2026-08-04 18:00:46.663749936 +0000 UTC m=+72.008151092"
Aug 04 18:00:46.665598 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.665566    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:46.681382 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.681331    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh" podStartSLOduration=18.538373082 podStartE2EDuration="24.681310722s" totalImagesPullingTime="6.14293764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:46.680531141 +0000 UTC m=+72.024932298" watchObservedRunningTime="2026-08-04 18:00:46.681310722 +0000 UTC m=+72.025711879"
Aug 04 18:00:46.722472 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:46.722427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.022533 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:47.022449    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-xgggh"
Aug 04 18:00:47.032594 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:47.032565    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.233906 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:47.233873    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.334506 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:47.334464    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.435007 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:47.434969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.535416 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:47.535382    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.635225 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:47.635137    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.735815 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:47.735781    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.892787 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:47.892702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:48.228296 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:48.228200    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:48.333082 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:48.333025    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:48.433682 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:48.433645    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:48.499062 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:48.498945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:48.628683 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:48.628649    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:48.935566 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:48.935524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:49.125962 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:49.125928    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:49.226296 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:49.226203    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:49.327425 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:49.327387    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:49.746268 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:49.746216    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:49.846978 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:49.846945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:54.195949 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.195893    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"]
Aug 04 18:00:54.198865 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.198846    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.201791 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.201758    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 04 18:00:54.202517 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.202489    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 04 18:00:54.202662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.202536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Aug 04 18:00:54.202662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.202568    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Aug 04 18:00:54.202662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.202536    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 04 18:00:54.203461 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.203446    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Aug 04 18:00:54.203530 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.203449    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Aug 04 18:00:54.212880 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.212848    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"]
Aug 04 18:00:54.292427 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.292384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-hub\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.292427 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.292422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.292680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.292481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.292680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.292535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxxd\" (UniqueName: \"kubernetes.io/projected/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-kube-api-access-2wxxd\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.292680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.292564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-ca\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.292680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.292597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.393271 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.393232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.393421 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.393349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-hub\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.393421 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.393374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.393559 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.393534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.393648 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.393629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxxd\" (UniqueName: \"kubernetes.io/projected/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-kube-api-access-2wxxd\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.393713 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.393672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-ca\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.394309 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.394225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.396306 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.396271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.396306 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.396292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-ca\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.396473 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.396324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-hub\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.396473 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.396443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.404422 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.404393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxxd\" (UniqueName: \"kubernetes.io/projected/d7f5fc3b-248d-4a4a-8ce2-be766656bedb-kube-api-access-2wxxd\") pod \"cluster-proxy-proxy-agent-f8c9d5f5-4dbt4\" (UID: \"d7f5fc3b-248d-4a4a-8ce2-be766656bedb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.514667 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.514582    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"
Aug 04 18:00:54.657031 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.657004    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4"]
Aug 04 18:00:54.659823 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:54.659795    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7f5fc3b_248d_4a4a_8ce2_be766656bedb.slice/crio-44817da3181d0a3e656c7924611285e619976948a5d16384b7a009c4dfdf5790 WatchSource:0}: Error finding container 44817da3181d0a3e656c7924611285e619976948a5d16384b7a009c4dfdf5790: Status 404 returned error can't find the container with id 44817da3181d0a3e656c7924611285e619976948a5d16384b7a009c4dfdf5790
Aug 04 18:00:54.897956 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.897911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:54.900564 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.900540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2335dcc1-9d52-483e-b1c6-1e58887ec4ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-wvj9z\" (UID: \"2335dcc1-9d52-483e-b1c6-1e58887ec4ca\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:54.998460 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.998425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:54.998460 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.998462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:54.998681 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.998493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:54.999141 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:54.999117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db104465-bd2b-4963-b19c-26f2b8438865-service-ca-bundle\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:55.000992 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.000972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db104465-bd2b-4963-b19c-26f2b8438865-metrics-certs\") pod \"router-default-785c67bddb-zlhb4\" (UID: \"db104465-bd2b-4963-b19c-26f2b8438865\") " pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:55.001096 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.000998    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37559df0-a4d3-48ac-87be-21de5499def8-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-tlnn6\" (UID: \"37559df0-a4d3-48ac-87be-21de5499def8\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:55.033657 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.033621    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-flkwk\""
Aug 04 18:00:55.041529 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.041500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"
Aug 04 18:00:55.142814 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.142785    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7b75h\""
Aug 04 18:00:55.150870 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.150779    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:55.178409 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.178370    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z"]
Aug 04 18:00:55.180862 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:55.180831    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2335dcc1_9d52_483e_b1c6_1e58887ec4ca.slice/crio-a9c110b0518bedfc325f92fbdb28dcd1da2ac0c7e3eec5a5f21c96f18c274357 WatchSource:0}: Error finding container a9c110b0518bedfc325f92fbdb28dcd1da2ac0c7e3eec5a5f21c96f18c274357: Status 404 returned error can't find the container with id a9c110b0518bedfc325f92fbdb28dcd1da2ac0c7e3eec5a5f21c96f18c274357
Aug 04 18:00:55.185713 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.185693    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-86gq6\""
Aug 04 18:00:55.194519 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.194420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"
Aug 04 18:00:55.319745 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.319715    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-785c67bddb-zlhb4"]
Aug 04 18:00:55.325246 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:55.325212    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb104465_bd2b_4963_b19c_26f2b8438865.slice/crio-615bf58a4070494ceb94b227cdb0af65bcdf56b37d12310206e3138f8d731b0b WatchSource:0}: Error finding container 615bf58a4070494ceb94b227cdb0af65bcdf56b37d12310206e3138f8d731b0b: Status 404 returned error can't find the container with id 615bf58a4070494ceb94b227cdb0af65bcdf56b37d12310206e3138f8d731b0b
Aug 04 18:00:55.369427 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.369399    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6"]
Aug 04 18:00:55.372932 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:00:55.372904    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37559df0_a4d3_48ac_87be_21de5499def8.slice/crio-08efbb6dc7587920db8cff66d8a2cb3b2790beadfaf033d7bdb6db98dc2bb62a WatchSource:0}: Error finding container 08efbb6dc7587920db8cff66d8a2cb3b2790beadfaf033d7bdb6db98dc2bb62a: Status 404 returned error can't find the container with id 08efbb6dc7587920db8cff66d8a2cb3b2790beadfaf033d7bdb6db98dc2bb62a
Aug 04 18:00:55.549477 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.549432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:55.641895 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.641855    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:55.666032 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.665992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z" event={"ID":"2335dcc1-9d52-483e-b1c6-1e58887ec4ca","Type":"ContainerStarted","Data":"a9c110b0518bedfc325f92fbdb28dcd1da2ac0c7e3eec5a5f21c96f18c274357"}
Aug 04 18:00:55.667209 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.667180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6" event={"ID":"37559df0-a4d3-48ac-87be-21de5499def8","Type":"ContainerStarted","Data":"08efbb6dc7587920db8cff66d8a2cb3b2790beadfaf033d7bdb6db98dc2bb62a"}
Aug 04 18:00:55.668937 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.668907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-785c67bddb-zlhb4" event={"ID":"db104465-bd2b-4963-b19c-26f2b8438865","Type":"ContainerStarted","Data":"3f161e1b33e442612f97f38b8e2a84b7e102e7bf83b918c8c57bd592c46527c8"}
Aug 04 18:00:55.669091 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.668955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-785c67bddb-zlhb4" event={"ID":"db104465-bd2b-4963-b19c-26f2b8438865","Type":"ContainerStarted","Data":"615bf58a4070494ceb94b227cdb0af65bcdf56b37d12310206e3138f8d731b0b"}
Aug 04 18:00:55.670716 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.670691    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4" event={"ID":"d7f5fc3b-248d-4a4a-8ce2-be766656bedb","Type":"ContainerStarted","Data":"44817da3181d0a3e656c7924611285e619976948a5d16384b7a009c4dfdf5790"}
Aug 04 18:00:55.690557 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.690497    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-785c67bddb-zlhb4" podStartSLOduration=33.690479446 podStartE2EDuration="33.690479446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:55.689631936 +0000 UTC m=+81.034033092" watchObservedRunningTime="2026-08-04 18:00:55.690479446 +0000 UTC m=+81.034880627"
Aug 04 18:00:55.790441 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.790302    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:55.850214 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:55.850116    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:56.151316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:56.151219    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:56.154449 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:56.154420    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:56.640344 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:56.640315    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5x45c"
Aug 04 18:00:56.675935 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:56.675907    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:56.678641 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:56.678616    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-785c67bddb-zlhb4"
Aug 04 18:00:57.682287 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:57.681935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6" event={"ID":"37559df0-a4d3-48ac-87be-21de5499def8","Type":"ContainerStarted","Data":"72346bcbfb2fde21748c465a46d3293ee821ae74d07d2ccc7c6c792850ec8d92"}
Aug 04 18:00:57.698745 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:57.698689    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-tlnn6" podStartSLOduration=33.429171858 podStartE2EDuration="34.698665187s" totalImagesPullingTime="1.269493329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:57.69807197 +0000 UTC m=+83.042473128" watchObservedRunningTime="2026-08-04 18:00:57.698665187 +0000 UTC m=+83.043066374"
Aug 04 18:00:57.816159 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:57.816095    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:57.882335 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:57.882231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:59.617459 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:59.617414    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:59.676685 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:00:59.676642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:00.494495 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:00.494398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:00.597281 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:00.597247    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:00.683083 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:00.683025    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:00.702296 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:00.702021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z" event={"ID":"2335dcc1-9d52-483e-b1c6-1e58887ec4ca","Type":"ContainerStarted","Data":"9c90ff98f63a78be78aeda45f5742bcbd83e52ec5da5d8bbaec51e5649c8cb9f"}
Aug 04 18:01:00.703867 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:00.703840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4" event={"ID":"d7f5fc3b-248d-4a4a-8ce2-be766656bedb","Type":"ContainerStarted","Data":"4861c2d620c0b03bb21c7d140b5917f5f34928dd713b53a94f2fbc3e729b76b0"}
Aug 04 18:01:00.719560 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:00.719506    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-wvj9z" podStartSLOduration=33.778953025 podStartE2EDuration="38.71948932s" totalImagesPullingTime="4.940536295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:00.718840752 +0000 UTC m=+86.063241909" watchObservedRunningTime="2026-08-04 18:01:00.71948932 +0000 UTC m=+86.063890476"
Aug 04 18:01:00.784803 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:00.784765    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:01.713307 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:01.713270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4" event={"ID":"d7f5fc3b-248d-4a4a-8ce2-be766656bedb","Type":"ContainerStarted","Data":"43190601b43db4d4549c2cbfcc2a8968c8df86485523ac70140e8895a5790d01"}
Aug 04 18:01:01.713307 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:01.713318    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4" event={"ID":"d7f5fc3b-248d-4a4a-8ce2-be766656bedb","Type":"ContainerStarted","Data":"bd2798c8daba43e08470f7947638ca1afb1a37057d961affb7d16ed6b131a417"}
Aug 04 18:01:01.733169 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:01.733112    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f8c9d5f5-4dbt4" podStartSLOduration=1.856064419 podStartE2EDuration="7.733095488s" totalImagesPullingTime="5.877031069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:01.731298478 +0000 UTC m=+87.075699626" watchObservedRunningTime="2026-08-04 18:01:01.733095488 +0000 UTC m=+87.077496646"
Aug 04 18:01:02.373191 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:02.373144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:01:02.375669 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:02.375641    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87611d06-12e3-4b27-8ce5-a156441f8b9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f767j\" (UID: \"87611d06-12e3-4b27-8ce5-a156441f8b9e\") " pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:01:02.651375 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:02.651285    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9fk8\""
Aug 04 18:01:02.659447 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:02.659418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f767j"
Aug 04 18:01:02.824068 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:02.824025    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f767j"]
Aug 04 18:01:02.826835 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:01:02.826803    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87611d06_12e3_4b27_8ce5_a156441f8b9e.slice/crio-a29b387e9c47f30c05f6177c8263acc068f3810de46ef14b19751105e642f3b7 WatchSource:0}: Error finding container a29b387e9c47f30c05f6177c8263acc068f3810de46ef14b19751105e642f3b7: Status 404 returned error can't find the container with id a29b387e9c47f30c05f6177c8263acc068f3810de46ef14b19751105e642f3b7
Aug 04 18:01:02.949072 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:02.949009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.049193 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:03.049161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.154396 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:03.154362    2581 patch_prober.go:28] interesting pod/image-registry-565b8c87c4-9gpcq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Aug 04 18:01:03.154567 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:03.154416    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq" podUID="ecebf12c-a9cc-44dd-bee5-d210c3209aa9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:01:03.452910 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:03.452879    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.511291 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:03.511258    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.567800 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:03.567767    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.715607 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:03.715515    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.722834 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:03.722806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f767j" event={"ID":"87611d06-12e3-4b27-8ce5-a156441f8b9e","Type":"ContainerStarted","Data":"0d7fad12bf2eb06be98d6c62548cad518e10b3e4fef029f459314036f92459ee"}
Aug 04 18:01:03.722994 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:03.722868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f767j" event={"ID":"87611d06-12e3-4b27-8ce5-a156441f8b9e","Type":"ContainerStarted","Data":"a29b387e9c47f30c05f6177c8263acc068f3810de46ef14b19751105e642f3b7"}
Aug 04 18:01:03.788387 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:03.788350    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.872387 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:03.872355    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.463196 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:04.463099    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.564206 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:04.564166    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.626215 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:04.626183    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:01:04.630213 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:04.630184    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.729141 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:04.729031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f767j" event={"ID":"87611d06-12e3-4b27-8ce5-a156441f8b9e","Type":"ContainerStarted","Data":"94b1cb9b31251fa8cd56d1c786ba5f8228d85500a3f8b6606f7c9de7c981cd66"}
Aug 04 18:01:04.860005 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:04.859965    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.958584 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:04.958549    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:05.049970 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:05.049913    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:05.734156 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:05.734070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f767j" event={"ID":"87611d06-12e3-4b27-8ce5-a156441f8b9e","Type":"ContainerStarted","Data":"badc4aec218675459a04807a075d851ba9baf8aec8447d018b19d90795b14960"}
Aug 04 18:01:05.754004 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:05.753937    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f767j" podStartSLOduration=33.407550399 podStartE2EDuration="35.753919343s" totalImagesPullingTime="2.346368944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:05.752215323 +0000 UTC m=+91.096616482" watchObservedRunningTime="2026-08-04 18:01:05.753919343 +0000 UTC m=+91.098320499"
Aug 04 18:01:08.150464 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:08.150427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:08.314749 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:08.314715    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:09.589142 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:09.589105    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:09.695408 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:09.695361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:09.876099 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:09.876006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.010960 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.010922    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.122545 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.122503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.161485 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.161402    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-48q26"]
Aug 04 18:01:10.164216 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.164194    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.166394 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.166370    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 04 18:01:10.166609 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.166592    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 04 18:01:10.166931 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.166909    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ltd8z\""
Aug 04 18:01:10.167033 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.166939    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 04 18:01:10.167444 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.167426    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 04 18:01:10.223849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.223811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.239972 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.239932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-metrics-client-ca\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.240183 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.240015    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-wtmp\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.240183 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.240062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-textfile\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.240183 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.240099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.240183 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.240146    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-root\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.240183 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.240172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-sys\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.240395 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.240198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9x8\" (UniqueName: \"kubernetes.io/projected/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-kube-api-access-cp9x8\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.240395 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.240314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.240395 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.240366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-tls\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.314143 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.314108    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.341190 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-metrics-client-ca\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341364 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-wtmp\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341364 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-textfile\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341364 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341523 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-wtmp\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341523 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-root\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341523 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-sys\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341523 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-root\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341523 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9x8\" (UniqueName: \"kubernetes.io/projected/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-kube-api-access-cp9x8\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341523 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-sys\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341790 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341790 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-tls\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341790 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-textfile\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341790 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:01:10.341731    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Aug 04 18:01:10.341942 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.341804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-metrics-client-ca\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.341942 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:01:10.341827    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-tls podName:c49f0a35-6d58-4699-94c2-b2ad4baf96ee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:10.84180692 +0000 UTC m=+96.186208055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-tls") pod "node-exporter-48q26" (UID: "c49f0a35-6d58-4699-94c2-b2ad4baf96ee") : secret "node-exporter-tls" not found
Aug 04 18:01:10.342035 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.342018    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.343861 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.343843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.354599 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.354570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9x8\" (UniqueName: \"kubernetes.io/projected/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-kube-api-access-cp9x8\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.447392 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.447300    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.548070 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.547998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.651451 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.651413    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.749740 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.749642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.845930 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.845894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-tls\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.848455 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.848431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c49f0a35-6d58-4699-94c2-b2ad4baf96ee-node-exporter-tls\") pod \"node-exporter-48q26\" (UID: \"c49f0a35-6d58-4699-94c2-b2ad4baf96ee\") " pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:10.877493 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:10.877440    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:11.074162 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:11.074126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-48q26"
Aug 04 18:01:11.085602 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:01:11.085574    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49f0a35_6d58_4699_94c2_b2ad4baf96ee.slice/crio-49fda63cd3b34be863c80c618e0eea776ff9bacc58e5dc6fe061eb6acfc38423 WatchSource:0}: Error finding container 49fda63cd3b34be863c80c618e0eea776ff9bacc58e5dc6fe061eb6acfc38423: Status 404 returned error can't find the container with id 49fda63cd3b34be863c80c618e0eea776ff9bacc58e5dc6fe061eb6acfc38423
Aug 04 18:01:11.168417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:11.168379    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:11.275955 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:11.275917    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:11.356356 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:11.356261    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:11.569552 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:11.569508    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:11.670934 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:11.670849    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:11.758465 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:11.758427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48q26" event={"ID":"c49f0a35-6d58-4699-94c2-b2ad4baf96ee","Type":"ContainerStarted","Data":"49fda63cd3b34be863c80c618e0eea776ff9bacc58e5dc6fe061eb6acfc38423"}
Aug 04 18:01:12.610892 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:12.610864    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gfgc5"
Aug 04 18:01:12.764067 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:12.764009    2581 generic.go:356] "Generic (PLEG): container finished" podID="c49f0a35-6d58-4699-94c2-b2ad4baf96ee" containerID="adf3eebee9d21fa0439ab3105dc9bcc9941f33909d43aa3dc7a7089fd95809d9" exitCode=0
Aug 04 18:01:12.764411 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:12.764069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48q26" event={"ID":"c49f0a35-6d58-4699-94c2-b2ad4baf96ee","Type":"ContainerDied","Data":"adf3eebee9d21fa0439ab3105dc9bcc9941f33909d43aa3dc7a7089fd95809d9"}
Aug 04 18:01:13.121055 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:13.120949    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:13.220532 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:13.220496    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:13.323410 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:13.323373    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:13.548700 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:13.548665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:13.716355 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:13.716309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:13.769781 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:13.769745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48q26" event={"ID":"c49f0a35-6d58-4699-94c2-b2ad4baf96ee","Type":"ContainerStarted","Data":"f269eddb7fe27153e94fac98d8ef46e630357f50eb2223ea367450ee4f8705d1"}
Aug 04 18:01:13.770297 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:13.769802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48q26" event={"ID":"c49f0a35-6d58-4699-94c2-b2ad4baf96ee","Type":"ContainerStarted","Data":"c0f503465215d4c98f573ece41b3266f246f0a078620e5a178eea6292e5b5e0b"}
Aug 04 18:01:13.793715 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:13.793665    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-48q26" podStartSLOduration=2.780303326 podStartE2EDuration="3.79364689s" totalImagesPullingTime="938.833522ms" totalInitContainerRuntime="74.510042ms" isStatefulPod=true podCreationTimestamp="2026-08-04 18:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:13.791847284 +0000 UTC m=+99.136248442" watchObservedRunningTime="2026-08-04 18:01:13.79364689 +0000 UTC m=+99.138048088"
Aug 04 18:01:13.798324 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:13.798284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:14.916970 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:14.916938    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:15.010780 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:15.010685    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:15.111867 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:15.111821    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:15.769230 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:15.769195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:15.867297 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:15.867236    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:16.468710 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:16.468673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:16.503124 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:16.503092    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-565b8c87c4-9gpcq"]
Aug 04 18:01:16.581301 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:16.581268    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:16.680986 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:16.680957    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:17.781748 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:17.781707    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:17.849579 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:17.849539    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:19.150832 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:19.150793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:19.222988 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:19.222957    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:22.566655 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:22.566621    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:22.666235 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:22.666195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:23.375035 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:23.374998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:23.515583 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:23.515535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:23.617171 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:23.617136    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:26.146751 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:26.146716    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:26.213607 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:26.213574    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:31.829018 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:31.828978    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:31.927977 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:31.927945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:32.027325 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:32.027289    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:34.134401 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:34.134358    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:34.203684 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:34.203640    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:35.858628 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:35.858593    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e7b1bac-ba42-4f89-bc21-b06342fec05c" containerID="fe48117984df7ec50ecbbf438da53b98a1639174bf947ce232ca3f0a462359e0" exitCode=0
Aug 04 18:01:35.859001 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:35.858666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9" event={"ID":"0e7b1bac-ba42-4f89-bc21-b06342fec05c","Type":"ContainerDied","Data":"fe48117984df7ec50ecbbf438da53b98a1639174bf947ce232ca3f0a462359e0"}
Aug 04 18:01:35.859059 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:35.859011    2581 scope.go:117] "RemoveContainer" containerID="fe48117984df7ec50ecbbf438da53b98a1639174bf947ce232ca3f0a462359e0"
Aug 04 18:01:36.864879 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:36.864842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-xftn9" event={"ID":"0e7b1bac-ba42-4f89-bc21-b06342fec05c","Type":"ContainerStarted","Data":"26fa897b9467ba3920b72df7d4a1227e1f3998898a9f84ee867034a412a84a7b"}
Aug 04 18:01:40.317398 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:40.317366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:40.418330 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:40.418298    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:41.524513 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.524474    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq" podUID="ecebf12c-a9cc-44dd-bee5-d210c3209aa9" containerName="registry" containerID="cri-o://3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb" gracePeriod=30
Aug 04 18:01:41.773257 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.773230    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:01:41.884938 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.884840    2581 generic.go:356] "Generic (PLEG): container finished" podID="ecebf12c-a9cc-44dd-bee5-d210c3209aa9" containerID="3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb" exitCode=0
Aug 04 18:01:41.884938 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.884892    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq"
Aug 04 18:01:41.885205 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.884934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq" event={"ID":"ecebf12c-a9cc-44dd-bee5-d210c3209aa9","Type":"ContainerDied","Data":"3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb"}
Aug 04 18:01:41.885205 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.884990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565b8c87c4-9gpcq" event={"ID":"ecebf12c-a9cc-44dd-bee5-d210c3209aa9","Type":"ContainerDied","Data":"52bfbae8df8b6dfe09e4d22697027592cc136fd782c8f626b85feed48af2d8db"}
Aug 04 18:01:41.885205 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.885016    2581 scope.go:117] "RemoveContainer" containerID="3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb"
Aug 04 18:01:41.893321 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.893301    2581 scope.go:117] "RemoveContainer" containerID="3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb"
Aug 04 18:01:41.893657 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:01:41.893635    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb\": container with ID starting with 3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb not found: ID does not exist" containerID="3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb"
Aug 04 18:01:41.893706 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.893670    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb"} err="failed to get container status \"3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb\": rpc error: code = NotFound desc = could not find container \"3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb\": container with ID starting with 3144bfb3b26eb64308c628c013d7b5699c73d1df19994514983103f7b8cd3cdb not found: ID does not exist"
Aug 04 18:01:41.915057 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.915002    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls\") pod \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") "
Aug 04 18:01:41.915194 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.915098    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-certificates\") pod \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") "
Aug 04 18:01:41.915194 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.915129    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx7jv\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-kube-api-access-wx7jv\") pod \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") "
Aug 04 18:01:41.915194 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.915154    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-bound-sa-token\") pod \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") "
Aug 04 18:01:41.915345 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.915194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-image-registry-private-configuration\") pod \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") "
Aug 04 18:01:41.915345 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.915223    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-trusted-ca\") pod \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") "
Aug 04 18:01:41.915345 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.915264    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-installation-pull-secrets\") pod \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") "
Aug 04 18:01:41.915345 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.915310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-ca-trust-extracted\") pod \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\" (UID: \"ecebf12c-a9cc-44dd-bee5-d210c3209aa9\") "
Aug 04 18:01:41.916457 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.916374    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ecebf12c-a9cc-44dd-bee5-d210c3209aa9" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:41.916457 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.916415    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ecebf12c-a9cc-44dd-bee5-d210c3209aa9" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:41.922098 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.919309    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ecebf12c-a9cc-44dd-bee5-d210c3209aa9" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:41.922098 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.919588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ecebf12c-a9cc-44dd-bee5-d210c3209aa9" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:41.922698 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.922660    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-kube-api-access-wx7jv" (OuterVolumeSpecName: "kube-api-access-wx7jv") pod "ecebf12c-a9cc-44dd-bee5-d210c3209aa9" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9"). InnerVolumeSpecName "kube-api-access-wx7jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:41.923127 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.923079    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ecebf12c-a9cc-44dd-bee5-d210c3209aa9" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:41.923466 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.923443    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ecebf12c-a9cc-44dd-bee5-d210c3209aa9" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:41.929515 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:41.929465    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ecebf12c-a9cc-44dd-bee5-d210c3209aa9" (UID: "ecebf12c-a9cc-44dd-bee5-d210c3209aa9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:01:42.016898 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:42.016847    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-certificates\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:01:42.016898 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:42.016893    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx7jv\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-kube-api-access-wx7jv\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:01:42.016898 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:42.016904    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-bound-sa-token\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:01:42.016898 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:42.016915    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-image-registry-private-configuration\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:01:42.017206 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:42.016926    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-trusted-ca\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:01:42.017206 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:42.016934    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-installation-pull-secrets\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:01:42.017206 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:42.016942    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-ca-trust-extracted\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:01:42.017206 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:42.016950    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecebf12c-a9cc-44dd-bee5-d210c3209aa9-registry-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:01:42.208389 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:42.208357    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-565b8c87c4-9gpcq"]
Aug 04 18:01:42.213133 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:42.213096    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-565b8c87c4-9gpcq"]
Aug 04 18:01:43.214618 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:43.214585    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecebf12c-a9cc-44dd-bee5-d210c3209aa9" path="/var/lib/kubelet/pods/ecebf12c-a9cc-44dd-bee5-d210c3209aa9/volumes"
Aug 04 18:01:45.044947 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:45.044906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:01:45.047496 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:45.047466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e86da9-f3b0-4819-872b-8f4944af85e8-metrics-certs\") pod \"network-metrics-daemon-gdpj9\" (UID: \"56e86da9-f3b0-4819-872b-8f4944af85e8\") " pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:01:45.127379 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:45.127340    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5s4h\""
Aug 04 18:01:45.135451 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:45.135425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdpj9"
Aug 04 18:01:45.282904 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:45.282868    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gdpj9"]
Aug 04 18:01:45.286478 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:01:45.286445    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e86da9_f3b0_4819_872b_8f4944af85e8.slice/crio-16634fc456c9b9dd85d7f06791450465a81580a094dfb585a68f7a733022a482 WatchSource:0}: Error finding container 16634fc456c9b9dd85d7f06791450465a81580a094dfb585a68f7a733022a482: Status 404 returned error can't find the container with id 16634fc456c9b9dd85d7f06791450465a81580a094dfb585a68f7a733022a482
Aug 04 18:01:45.902430 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:45.902397    2581 generic.go:356] "Generic (PLEG): container finished" podID="d4962250-4950-463c-92ee-b6d105f03d80" containerID="4929e8de6e21c4a67f00d98f622361a07f6c8a624ca5a51bd0e8c1ff9c163b93" exitCode=0
Aug 04 18:01:45.902631 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:45.902471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz" event={"ID":"d4962250-4950-463c-92ee-b6d105f03d80","Type":"ContainerDied","Data":"4929e8de6e21c4a67f00d98f622361a07f6c8a624ca5a51bd0e8c1ff9c163b93"}
Aug 04 18:01:45.902850 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:45.902834    2581 scope.go:117] "RemoveContainer" containerID="4929e8de6e21c4a67f00d98f622361a07f6c8a624ca5a51bd0e8c1ff9c163b93"
Aug 04 18:01:45.903809 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:45.903777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdpj9" event={"ID":"56e86da9-f3b0-4819-872b-8f4944af85e8","Type":"ContainerStarted","Data":"16634fc456c9b9dd85d7f06791450465a81580a094dfb585a68f7a733022a482"}
Aug 04 18:01:46.909631 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:46.909584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-pfvlz" event={"ID":"d4962250-4950-463c-92ee-b6d105f03d80","Type":"ContainerStarted","Data":"e9ac7fbbd21682c234987ea180b06749a8b48f23becd989f99f91ff864ac93be"}
Aug 04 18:01:46.911323 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:46.911293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdpj9" event={"ID":"56e86da9-f3b0-4819-872b-8f4944af85e8","Type":"ContainerStarted","Data":"b7620b63daee6c8d00a6ba6fa80bc99fabf62157503a59857daf5e5b621ff35e"}
Aug 04 18:01:46.911447 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:46.911337    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdpj9" event={"ID":"56e86da9-f3b0-4819-872b-8f4944af85e8","Type":"ContainerStarted","Data":"4d8e53954521d5975660e42ba0c7559b2ff8d2248fee5f3482129c9327316c72"}
Aug 04 18:01:46.945751 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:46.945696    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gdpj9" podStartSLOduration=129.725884439 podStartE2EDuration="2m10.945674165s" totalImagesPullingTime="1.219789726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:46.944506036 +0000 UTC m=+132.288907198" watchObservedRunningTime="2026-08-04 18:01:46.945674165 +0000 UTC m=+132.290075323"
Aug 04 18:01:47.072369 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:47.072332    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:47.153125 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:47.153086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:50.927360 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:50.927327    2581 generic.go:356] "Generic (PLEG): container finished" podID="7e8f66e5-a4db-45ee-877a-de0246618be7" containerID="2ce3bc8192d881ecfb16d7e107bfaaffa14d93f589d40ccd6a608a8a252731a5" exitCode=0
Aug 04 18:01:50.927785 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:50.927396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-zxjxn" event={"ID":"7e8f66e5-a4db-45ee-877a-de0246618be7","Type":"ContainerDied","Data":"2ce3bc8192d881ecfb16d7e107bfaaffa14d93f589d40ccd6a608a8a252731a5"}
Aug 04 18:01:50.927785 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:50.927757    2581 scope.go:117] "RemoveContainer" containerID="2ce3bc8192d881ecfb16d7e107bfaaffa14d93f589d40ccd6a608a8a252731a5"
Aug 04 18:01:51.933503 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:01:51.933467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-zxjxn" event={"ID":"7e8f66e5-a4db-45ee-877a-de0246618be7","Type":"ContainerStarted","Data":"13aabbc81ddef876ceaf9586c5b59155857c2df25760dfdf49880619fc2cdb48"}
Aug 04 18:02:26.362945 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:02:26.362860    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:26.460671 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:02:26.460634    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:38.342372 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:02:38.342336    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:38.468624 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:02:38.468586    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:44.487767 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:02:44.487732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:49.158978 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:02:49.158945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:49.263216 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:02:49.263178    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:49.329322 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:02:49.329283    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:55.099537 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:02:55.099507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:55.196755 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:02:55.196718    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:55.720202 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:02:55.720163    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:32.360189 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:03:32.360148    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:35.113722 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:04:35.113693    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:04:35.119275 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:04:35.119246    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:04:35.121907 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:04:35.121880    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:04:35.125758 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:04:35.125730    2581 kubelet.go:1648] "Image garbage collection succeeded"
Aug 04 18:04:35.127906 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:04:35.127882    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:05:30.846532 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:05:30.846425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:33.202350 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:05:33.202308    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:33.727999 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:05:33.727960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:33.828265 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:05:33.828222    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:41.299002 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:05:41.298967    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:41.397290 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:05:41.397257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:41.464906 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:05:41.464872    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:03.670957 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:03.670923    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:03.762814 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:03.762770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:09.256145 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:09.256110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:09.556586 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:09.556549    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:09.774718 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:09.774685    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:09.935775 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:09.935677    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:10.100431 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.100400    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-dmsvh"]
Aug 04 18:06:10.100679 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.100668    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecebf12c-a9cc-44dd-bee5-d210c3209aa9" containerName="registry"
Aug 04 18:06:10.100721 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.100681    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecebf12c-a9cc-44dd-bee5-d210c3209aa9" containerName="registry"
Aug 04 18:06:10.100773 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.100764    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecebf12c-a9cc-44dd-bee5-d210c3209aa9" containerName="registry"
Aug 04 18:06:10.103024 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.103006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-dmsvh"
Aug 04 18:06:10.105231 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.105199    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Aug 04 18:06:10.105380 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.105366    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Aug 04 18:06:10.106198 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.106180    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-57k9r\""
Aug 04 18:06:10.106297 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.106204    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Aug 04 18:06:10.106297 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.106269    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Aug 04 18:06:10.114578 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.114553    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-dmsvh"]
Aug 04 18:06:10.195815 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.195721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mnz4\" (UniqueName: \"kubernetes.io/projected/8faffece-ac58-4894-997b-0709df38ee1f-kube-api-access-2mnz4\") pod \"keda-admission-56f9798bb4-dmsvh\" (UID: \"8faffece-ac58-4894-997b-0709df38ee1f\") " pod="openshift-keda/keda-admission-56f9798bb4-dmsvh"
Aug 04 18:06:10.195815 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.195790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8faffece-ac58-4894-997b-0709df38ee1f-certificates\") pod \"keda-admission-56f9798bb4-dmsvh\" (UID: \"8faffece-ac58-4894-997b-0709df38ee1f\") " pod="openshift-keda/keda-admission-56f9798bb4-dmsvh"
Aug 04 18:06:10.296882 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.296839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8faffece-ac58-4894-997b-0709df38ee1f-certificates\") pod \"keda-admission-56f9798bb4-dmsvh\" (UID: \"8faffece-ac58-4894-997b-0709df38ee1f\") " pod="openshift-keda/keda-admission-56f9798bb4-dmsvh"
Aug 04 18:06:10.297276 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.296904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mnz4\" (UniqueName: \"kubernetes.io/projected/8faffece-ac58-4894-997b-0709df38ee1f-kube-api-access-2mnz4\") pod \"keda-admission-56f9798bb4-dmsvh\" (UID: \"8faffece-ac58-4894-997b-0709df38ee1f\") " pod="openshift-keda/keda-admission-56f9798bb4-dmsvh"
Aug 04 18:06:10.299560 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.299540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8faffece-ac58-4894-997b-0709df38ee1f-certificates\") pod \"keda-admission-56f9798bb4-dmsvh\" (UID: \"8faffece-ac58-4894-997b-0709df38ee1f\") " pod="openshift-keda/keda-admission-56f9798bb4-dmsvh"
Aug 04 18:06:10.308815 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.308785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mnz4\" (UniqueName: \"kubernetes.io/projected/8faffece-ac58-4894-997b-0709df38ee1f-kube-api-access-2mnz4\") pod \"keda-admission-56f9798bb4-dmsvh\" (UID: \"8faffece-ac58-4894-997b-0709df38ee1f\") " pod="openshift-keda/keda-admission-56f9798bb4-dmsvh"
Aug 04 18:06:10.413004 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.412962    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-dmsvh"
Aug 04 18:06:10.555482 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.555452    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-dmsvh"]
Aug 04 18:06:10.558221 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:06:10.558193    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8faffece_ac58_4894_997b_0709df38ee1f.slice/crio-d337408dcf1d4620ab151980fa3b8b559071cb5f72c47b41d075e30c1869d372 WatchSource:0}: Error finding container d337408dcf1d4620ab151980fa3b8b559071cb5f72c47b41d075e30c1869d372: Status 404 returned error can't find the container with id d337408dcf1d4620ab151980fa3b8b559071cb5f72c47b41d075e30c1869d372
Aug 04 18:06:10.559532 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.559515    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:06:10.894559 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:10.894520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-dmsvh" event={"ID":"8faffece-ac58-4894-997b-0709df38ee1f","Type":"ContainerStarted","Data":"d337408dcf1d4620ab151980fa3b8b559071cb5f72c47b41d075e30c1869d372"}
Aug 04 18:06:11.145434 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:11.145345    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:11.245806 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:11.245768    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:12.905881 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:12.905841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-dmsvh" event={"ID":"8faffece-ac58-4894-997b-0709df38ee1f","Type":"ContainerStarted","Data":"abaadb9f0976e95ca437ed1429a056b29dc8acce8da42de43647c9b32740e79c"}
Aug 04 18:06:12.906289 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:12.905952    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-dmsvh"
Aug 04 18:06:12.931766 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:12.931712    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-dmsvh" podStartSLOduration=0.712440251 podStartE2EDuration="2.931692547s" totalImagesPullingTime="2.219252296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:06:12.930370497 +0000 UTC m=+398.274771655" watchObservedRunningTime="2026-08-04 18:06:12.931692547 +0000 UTC m=+398.276093703"
Aug 04 18:06:33.912319 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:06:33.912284    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-dmsvh"
Aug 04 18:07:02.290022 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:02.289936    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:02.394592 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:02.394557    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:28.048725 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.048681    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:28.109572 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.109534    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"]
Aug 04 18:07:28.112692 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.112669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:07:28.115323 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.115286    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Aug 04 18:07:28.115323 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.115310    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-g8zh9\""
Aug 04 18:07:28.115323 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.115319    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Aug 04 18:07:28.116286 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.116261    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Aug 04 18:07:28.119019 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.118980    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:28.122951 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.122927    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"]
Aug 04 18:07:28.137357 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.137320    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-smfnx"]
Aug 04 18:07:28.140621 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.140602    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-smfnx"
Aug 04 18:07:28.142914 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.142888    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gqc29\""
Aug 04 18:07:28.143092 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.142989    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Aug 04 18:07:28.148771 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.148746    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-smfnx"]
Aug 04 18:07:28.199367 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.199337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6db6225-1429-464f-a8f5-411a899eac74-cert\") pod \"kserve-controller-manager-5f9d7f69b6-zr2h6\" (UID: \"a6db6225-1429-464f-a8f5-411a899eac74\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:07:28.199538 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.199382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2fz\" (UniqueName: \"kubernetes.io/projected/a6db6225-1429-464f-a8f5-411a899eac74-kube-api-access-hc2fz\") pod \"kserve-controller-manager-5f9d7f69b6-zr2h6\" (UID: \"a6db6225-1429-464f-a8f5-411a899eac74\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:07:28.199538 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.199417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/407cd864-f36b-4ec7-a59b-64fb9cb8b1d9-data\") pod \"seaweedfs-64568bcd49-smfnx\" (UID: \"407cd864-f36b-4ec7-a59b-64fb9cb8b1d9\") " pod="kserve/seaweedfs-64568bcd49-smfnx"
Aug 04 18:07:28.199538 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.199504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tshhn\" (UniqueName: \"kubernetes.io/projected/407cd864-f36b-4ec7-a59b-64fb9cb8b1d9-kube-api-access-tshhn\") pod \"seaweedfs-64568bcd49-smfnx\" (UID: \"407cd864-f36b-4ec7-a59b-64fb9cb8b1d9\") " pod="kserve/seaweedfs-64568bcd49-smfnx"
Aug 04 18:07:28.202597 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.202559    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:28.300195 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.300091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tshhn\" (UniqueName: \"kubernetes.io/projected/407cd864-f36b-4ec7-a59b-64fb9cb8b1d9-kube-api-access-tshhn\") pod \"seaweedfs-64568bcd49-smfnx\" (UID: \"407cd864-f36b-4ec7-a59b-64fb9cb8b1d9\") " pod="kserve/seaweedfs-64568bcd49-smfnx"
Aug 04 18:07:28.300195 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.300150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6db6225-1429-464f-a8f5-411a899eac74-cert\") pod \"kserve-controller-manager-5f9d7f69b6-zr2h6\" (UID: \"a6db6225-1429-464f-a8f5-411a899eac74\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:07:28.300195 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.300183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2fz\" (UniqueName: \"kubernetes.io/projected/a6db6225-1429-464f-a8f5-411a899eac74-kube-api-access-hc2fz\") pod \"kserve-controller-manager-5f9d7f69b6-zr2h6\" (UID: \"a6db6225-1429-464f-a8f5-411a899eac74\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:07:28.300407 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.300214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/407cd864-f36b-4ec7-a59b-64fb9cb8b1d9-data\") pod \"seaweedfs-64568bcd49-smfnx\" (UID: \"407cd864-f36b-4ec7-a59b-64fb9cb8b1d9\") " pod="kserve/seaweedfs-64568bcd49-smfnx"
Aug 04 18:07:28.300608 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.300589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/407cd864-f36b-4ec7-a59b-64fb9cb8b1d9-data\") pod \"seaweedfs-64568bcd49-smfnx\" (UID: \"407cd864-f36b-4ec7-a59b-64fb9cb8b1d9\") " pod="kserve/seaweedfs-64568bcd49-smfnx"
Aug 04 18:07:28.303025 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.302996    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:28.303221 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.303200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6db6225-1429-464f-a8f5-411a899eac74-cert\") pod \"kserve-controller-manager-5f9d7f69b6-zr2h6\" (UID: \"a6db6225-1429-464f-a8f5-411a899eac74\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:07:28.309421 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.309392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2fz\" (UniqueName: \"kubernetes.io/projected/a6db6225-1429-464f-a8f5-411a899eac74-kube-api-access-hc2fz\") pod \"kserve-controller-manager-5f9d7f69b6-zr2h6\" (UID: \"a6db6225-1429-464f-a8f5-411a899eac74\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:07:28.312159 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.312137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tshhn\" (UniqueName: \"kubernetes.io/projected/407cd864-f36b-4ec7-a59b-64fb9cb8b1d9-kube-api-access-tshhn\") pod \"seaweedfs-64568bcd49-smfnx\" (UID: \"407cd864-f36b-4ec7-a59b-64fb9cb8b1d9\") " pod="kserve/seaweedfs-64568bcd49-smfnx"
Aug 04 18:07:28.407966 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.407933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:28.424997 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.424961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:07:28.450634 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.450602    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-smfnx"
Aug 04 18:07:28.580109 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.580036    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"]
Aug 04 18:07:28.582418 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:07:28.582390    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6db6225_1429_464f_a8f5_411a899eac74.slice/crio-dec39f1b2189376376f84a0f05f84e43efb1c90d90493bea4f7198c7e1492643 WatchSource:0}: Error finding container dec39f1b2189376376f84a0f05f84e43efb1c90d90493bea4f7198c7e1492643: Status 404 returned error can't find the container with id dec39f1b2189376376f84a0f05f84e43efb1c90d90493bea4f7198c7e1492643
Aug 04 18:07:28.605436 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.605409    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-smfnx"]
Aug 04 18:07:28.607833 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:07:28.607811    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407cd864_f36b_4ec7_a59b_64fb9cb8b1d9.slice/crio-2552d774fa12ea017b49913756b48cf8d87c404f2b63d113f909e7c34931cb8e WatchSource:0}: Error finding container 2552d774fa12ea017b49913756b48cf8d87c404f2b63d113f909e7c34931cb8e: Status 404 returned error can't find the container with id 2552d774fa12ea017b49913756b48cf8d87c404f2b63d113f909e7c34931cb8e
Aug 04 18:07:28.637138 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.637100    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:28.737893 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.737860    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:28.805981 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:28.805862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:29.198249 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:29.198032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6" event={"ID":"a6db6225-1429-464f-a8f5-411a899eac74","Type":"ContainerStarted","Data":"dec39f1b2189376376f84a0f05f84e43efb1c90d90493bea4f7198c7e1492643"}
Aug 04 18:07:29.203522 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:29.203421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-smfnx" event={"ID":"407cd864-f36b-4ec7-a59b-64fb9cb8b1d9","Type":"ContainerStarted","Data":"2552d774fa12ea017b49913756b48cf8d87c404f2b63d113f909e7c34931cb8e"}
Aug 04 18:07:33.225721 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:33.225678    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6" event={"ID":"a6db6225-1429-464f-a8f5-411a899eac74","Type":"ContainerStarted","Data":"b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d"}
Aug 04 18:07:33.226193 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:33.225792    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:07:33.227316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:33.227293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-smfnx" event={"ID":"407cd864-f36b-4ec7-a59b-64fb9cb8b1d9","Type":"ContainerStarted","Data":"af382068e63d9fa404e161117c80eb507384ea3b5c828b1ca9ce9e120aac6b00"}
Aug 04 18:07:33.227436 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:33.227368    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-smfnx"
Aug 04 18:07:33.244735 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:33.244687    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6" podStartSLOduration=1.60489863 podStartE2EDuration="5.244671124s" totalImagesPullingTime="3.639772494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:07:33.244189003 +0000 UTC m=+478.588590410" watchObservedRunningTime="2026-08-04 18:07:33.244671124 +0000 UTC m=+478.589072336"
Aug 04 18:07:33.262171 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:33.262114    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-smfnx" podStartSLOduration=1.5558823579999999 podStartE2EDuration="5.262096989s" totalImagesPullingTime="3.706214631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:07:33.260342824 +0000 UTC m=+478.604743983" watchObservedRunningTime="2026-08-04 18:07:33.262096989 +0000 UTC m=+478.606498148"
Aug 04 18:07:39.233803 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:07:39.233769    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-smfnx"
Aug 04 18:08:04.237771 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:04.237726    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:08:10.716499 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.716460    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"]
Aug 04 18:08:10.717017 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.716677    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6" podUID="a6db6225-1429-464f-a8f5-411a899eac74" containerName="manager" containerID="cri-o://b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d" gracePeriod=10
Aug 04 18:08:10.739862 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.739832    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"]
Aug 04 18:08:10.742293 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.742275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"
Aug 04 18:08:10.759224 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.755237    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"]
Aug 04 18:08:10.855720 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.855673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqvt\" (UniqueName: \"kubernetes.io/projected/6251f072-33a3-47f5-b8da-33fe89b5b1e5-kube-api-access-zxqvt\") pod \"kserve-controller-manager-5f9d7f69b6-p7zxf\" (UID: \"6251f072-33a3-47f5-b8da-33fe89b5b1e5\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"
Aug 04 18:08:10.855917 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.855740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6251f072-33a3-47f5-b8da-33fe89b5b1e5-cert\") pod \"kserve-controller-manager-5f9d7f69b6-p7zxf\" (UID: \"6251f072-33a3-47f5-b8da-33fe89b5b1e5\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"
Aug 04 18:08:10.955398 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.955367    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:08:10.956292 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.956271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqvt\" (UniqueName: \"kubernetes.io/projected/6251f072-33a3-47f5-b8da-33fe89b5b1e5-kube-api-access-zxqvt\") pod \"kserve-controller-manager-5f9d7f69b6-p7zxf\" (UID: \"6251f072-33a3-47f5-b8da-33fe89b5b1e5\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"
Aug 04 18:08:10.956345 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.956320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6251f072-33a3-47f5-b8da-33fe89b5b1e5-cert\") pod \"kserve-controller-manager-5f9d7f69b6-p7zxf\" (UID: \"6251f072-33a3-47f5-b8da-33fe89b5b1e5\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"
Aug 04 18:08:10.958743 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.958722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6251f072-33a3-47f5-b8da-33fe89b5b1e5-cert\") pod \"kserve-controller-manager-5f9d7f69b6-p7zxf\" (UID: \"6251f072-33a3-47f5-b8da-33fe89b5b1e5\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"
Aug 04 18:08:10.965346 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:10.965311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqvt\" (UniqueName: \"kubernetes.io/projected/6251f072-33a3-47f5-b8da-33fe89b5b1e5-kube-api-access-zxqvt\") pod \"kserve-controller-manager-5f9d7f69b6-p7zxf\" (UID: \"6251f072-33a3-47f5-b8da-33fe89b5b1e5\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"
Aug 04 18:08:11.057339 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.057290    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6db6225-1429-464f-a8f5-411a899eac74-cert\") pod \"a6db6225-1429-464f-a8f5-411a899eac74\" (UID: \"a6db6225-1429-464f-a8f5-411a899eac74\") "
Aug 04 18:08:11.057507 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.057349    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc2fz\" (UniqueName: \"kubernetes.io/projected/a6db6225-1429-464f-a8f5-411a899eac74-kube-api-access-hc2fz\") pod \"a6db6225-1429-464f-a8f5-411a899eac74\" (UID: \"a6db6225-1429-464f-a8f5-411a899eac74\") "
Aug 04 18:08:11.059683 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.059642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6db6225-1429-464f-a8f5-411a899eac74-kube-api-access-hc2fz" (OuterVolumeSpecName: "kube-api-access-hc2fz") pod "a6db6225-1429-464f-a8f5-411a899eac74" (UID: "a6db6225-1429-464f-a8f5-411a899eac74"). InnerVolumeSpecName "kube-api-access-hc2fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:08:11.059683 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.059660    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6db6225-1429-464f-a8f5-411a899eac74-cert" (OuterVolumeSpecName: "cert") pod "a6db6225-1429-464f-a8f5-411a899eac74" (UID: "a6db6225-1429-464f-a8f5-411a899eac74"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:08:11.096864 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.096826    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"
Aug 04 18:08:11.158494 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.158456    2581 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6db6225-1429-464f-a8f5-411a899eac74-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:08:11.158494 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.158495    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2fz\" (UniqueName: \"kubernetes.io/projected/a6db6225-1429-464f-a8f5-411a899eac74-kube-api-access-hc2fz\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:08:11.236026 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.236000    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"]
Aug 04 18:08:11.238858 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:08:11.238828    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6251f072_33a3_47f5_b8da_33fe89b5b1e5.slice/crio-da1ecf0298ce07292ced7948a513a02347551d3cc7c630612019cd538abd81d5 WatchSource:0}: Error finding container da1ecf0298ce07292ced7948a513a02347551d3cc7c630612019cd538abd81d5: Status 404 returned error can't find the container with id da1ecf0298ce07292ced7948a513a02347551d3cc7c630612019cd538abd81d5
Aug 04 18:08:11.385241 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.385146    2581 generic.go:356] "Generic (PLEG): container finished" podID="a6db6225-1429-464f-a8f5-411a899eac74" containerID="b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d" exitCode=0
Aug 04 18:08:11.385241 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.385195    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"
Aug 04 18:08:11.385241 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.385230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6" event={"ID":"a6db6225-1429-464f-a8f5-411a899eac74","Type":"ContainerDied","Data":"b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d"}
Aug 04 18:08:11.385495 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.385282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-zr2h6" event={"ID":"a6db6225-1429-464f-a8f5-411a899eac74","Type":"ContainerDied","Data":"dec39f1b2189376376f84a0f05f84e43efb1c90d90493bea4f7198c7e1492643"}
Aug 04 18:08:11.385495 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.385303    2581 scope.go:117] "RemoveContainer" containerID="b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d"
Aug 04 18:08:11.386605 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.386584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf" event={"ID":"6251f072-33a3-47f5-b8da-33fe89b5b1e5","Type":"ContainerStarted","Data":"da1ecf0298ce07292ced7948a513a02347551d3cc7c630612019cd538abd81d5"}
Aug 04 18:08:11.393604 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.393579    2581 scope.go:117] "RemoveContainer" containerID="b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d"
Aug 04 18:08:11.393933 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:08:11.393913    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d\": container with ID starting with b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d not found: ID does not exist" containerID="b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d"
Aug 04 18:08:11.393991 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.393945    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d"} err="failed to get container status \"b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d\": rpc error: code = NotFound desc = could not find container \"b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d\": container with ID starting with b4af5fbb93782857e135b8592beec76270b0306fab3db3f8b8348a7f45ccd19d not found: ID does not exist"
Aug 04 18:08:11.404087 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.404037    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"]
Aug 04 18:08:11.410851 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:11.410815    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-zr2h6"]
Aug 04 18:08:12.392721 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:12.392688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf" event={"ID":"6251f072-33a3-47f5-b8da-33fe89b5b1e5","Type":"ContainerStarted","Data":"152fe93ede2cd815e1a73218961570550b7c1fe06cf782c836ab283a5e9ed1b8"}
Aug 04 18:08:12.393176 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:12.392805    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"
Aug 04 18:08:12.410554 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:12.410493    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf" podStartSLOduration=1.799674992 podStartE2EDuration="2.410473333s" totalImagesPullingTime="610.798341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:12.409585292 +0000 UTC m=+517.753986450" watchObservedRunningTime="2026-08-04 18:08:12.410473333 +0000 UTC m=+517.754874490"
Aug 04 18:08:13.214945 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:13.214908    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6db6225-1429-464f-a8f5-411a899eac74" path="/var/lib/kubelet/pods/a6db6225-1429-464f-a8f5-411a899eac74/volumes"
Aug 04 18:08:38.830407 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:38.830363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:43.402327 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:43.402288    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-p7zxf"
Aug 04 18:08:44.395748 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:44.395715    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:44.481349 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:44.481300    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:44.653844 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:44.653750    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:44.783413 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:44.783376    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:44.883099 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:44.883038    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:45.089523 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:45.089482    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:45.189632 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:08:45.189596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:10.095350 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:09:10.095311    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:35.139234 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:09:35.139204    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:09:35.143434 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:09:35.143407    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:09:35.146268 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:09:35.146243    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:09:35.150417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:09:35.150394    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:10:20.560002 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:10:20.559962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:40.718008 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:10:40.717973    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:43.577187 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:10:43.577152    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:43.736481 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:10:43.736442    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:56.384054 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:10:56.383998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:56.483798 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:10:56.483762    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:05.039673 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:11:05.039637    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:05.099253 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:11:05.099208    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:08.945810 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:11:08.945764    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:09.139255 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:11:09.139213    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:09.315311 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:11:09.315276    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:09.460338 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:11:09.460306    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:12.723211 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:11:12.723165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:13.463155 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:11:13.463115    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:15.537469 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:12:15.537421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:04.168130 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:13:04.168029    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:04.280999 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:13:04.280966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:04.754923 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:13:04.754878    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:07.793851 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:13:07.793815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:07.892372 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:13:07.892320    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:14.083391 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:13:14.083349    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:19.178129 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:13:19.178086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:19.585162 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:13:19.585125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:48.690602 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:13:48.690567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:35.163613 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:14:35.163521    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:14:35.164739 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:14:35.164714    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:14:35.171032 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:14:35.171009    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:14:35.171897 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:14:35.171879    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:14:39.203495 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:14:39.203459    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:39.306874 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:14:39.306827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:40.005074 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:14:40.005013    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:59.276911 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:14:59.276865    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:59.580889 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:14:59.580793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:04.326664 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:15:04.326616    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:04.719449 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:15:04.719351    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:22.446768 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:15:22.446724    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:22.508542 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:15:22.508509    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:27.632555 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:15:27.632522    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:27.857388 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:15:27.857354    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:37.663722 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:15:37.663679    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:42.833836 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:15:42.833792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:45.061894 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:15:45.061857    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:53.235589 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:15:53.235553    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:44.572090 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:16:44.572027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:44.653221 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:16:44.653191    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:45.046975 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:16:45.046945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:32.996514 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:17:32.996412    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:38.188826 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:17:38.188793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:38.450645 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:17:38.450551    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:18.388542 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:19:18.388466    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:23.433266 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:19:23.433215    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:23.750586 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:19:23.750494    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:35.188447 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:19:35.188417    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:19:35.188916 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:19:35.188487    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:19:35.195578 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:19:35.195551    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:19:35.195792 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:19:35.195776    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:19:38.815246 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:19:38.815213    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:51.302300 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:19:51.302255    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:56.422730 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:19:56.422697    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:56.647284 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:19:56.647251    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:53.626764 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:53.626710    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:58.712795 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.712764    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:58.750228 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.750191    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"]
Aug 04 18:20:58.750570 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.750552    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6db6225-1429-464f-a8f5-411a899eac74" containerName="manager"
Aug 04 18:20:58.750622 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.750573    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6db6225-1429-464f-a8f5-411a899eac74" containerName="manager"
Aug 04 18:20:58.750657 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.750626    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6db6225-1429-464f-a8f5-411a899eac74" containerName="manager"
Aug 04 18:20:58.753702 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.753684    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.755954 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.755934    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Aug 04 18:20:58.756101 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.756085    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:20:58.756168 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.756133    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:20:58.756224 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.756176    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:20:58.756364 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.756351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:20:58.756734 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.756718    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:20:58.756786 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.756748    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Aug 04 18:20:58.763409 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.763382    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"]
Aug 04 18:20:58.824732 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.824694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2hk\" (UniqueName: \"kubernetes.io/projected/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-api-access-hg2hk\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.824732 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.824735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.824934 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.824819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.824934 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.824849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.824998 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.824928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.875505 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.875469    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:58.926219 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.926180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2hk\" (UniqueName: \"kubernetes.io/projected/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-api-access-hg2hk\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.926437 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.926228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.926437 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.926272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.926437 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.926298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.926437 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.926330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.926785 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.926756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.927069 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.927026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.927142 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.927028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.929026 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.929006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:58.936005 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:58.935975    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2hk\" (UniqueName: \"kubernetes.io/projected/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-api-access-hg2hk\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:59.066067 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:59.066004    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:20:59.250180 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:59.250149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"]
Aug 04 18:20:59.253396 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:20:59.253368    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddd38af_ef8e_4c30_b936_0d0bc2ddaa4a.slice/crio-f230084e9ce1574f03553f341a25d32a8e21da6104075ca201ceb81c5ef9f483 WatchSource:0}: Error finding container f230084e9ce1574f03553f341a25d32a8e21da6104075ca201ceb81c5ef9f483: Status 404 returned error can't find the container with id f230084e9ce1574f03553f341a25d32a8e21da6104075ca201ceb81c5ef9f483
Aug 04 18:20:59.255243 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:59.255223    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:20:59.372310 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:20:59.372218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" event={"ID":"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a","Type":"ContainerStarted","Data":"f230084e9ce1574f03553f341a25d32a8e21da6104075ca201ceb81c5ef9f483"}
Aug 04 18:21:03.389901 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:03.389865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" event={"ID":"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a","Type":"ContainerStarted","Data":"e464fffd201bd3faf996249a0ec26d1663f9c490fdef280df67d8f372aa78c65"}
Aug 04 18:21:05.577705 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:05.577668    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:07.405887 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:07.405802    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerID="e464fffd201bd3faf996249a0ec26d1663f9c490fdef280df67d8f372aa78c65" exitCode=0
Aug 04 18:21:07.405887 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:07.405874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" event={"ID":"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a","Type":"ContainerDied","Data":"e464fffd201bd3faf996249a0ec26d1663f9c490fdef280df67d8f372aa78c65"}
Aug 04 18:21:29.524304 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:29.524268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" event={"ID":"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a","Type":"ContainerStarted","Data":"fae9e7e5ea8043e024cdd24a69f1b3abc764204831a167bf391da35c86747b26"}
Aug 04 18:21:32.544755 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:32.544713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" event={"ID":"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a","Type":"ContainerStarted","Data":"0d5c4b003a5456440f1eba2327c89fa2b690609d6c14dc24a23746135ee8dc84"}
Aug 04 18:21:33.548550 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:33.548513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:21:33.575986 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:33.575932    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" podStartSLOduration=2.904469327 podStartE2EDuration="35.575915823s" totalImagesPullingTime="28.822652628s" totalInitContainerRuntime="3.848793868s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:20:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:33.574392737 +0000 UTC m=+1318.918793909" watchObservedRunningTime="2026-08-04 18:21:33.575915823 +0000 UTC m=+1318.920316980"
Aug 04 18:21:34.552303 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:34.552275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:21:40.561755 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:40.561723    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:21:40.570880 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:40.570851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:40.663995 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:40.663957    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:40.741825 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:40.741794    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:41.272031 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:41.271999    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:41.356730 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:41.356701    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:51.634478 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:51.634425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:51.729954 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:51.729910    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:51.832929 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:21:51.832890    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:00.563400 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:00.563314    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:22:10.563008 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:10.562966    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:22:12.887507 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:12.887478    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:18.036057 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:18.035998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:18.145017 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:18.144938    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:20.562654 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:20.562561    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:22:30.562961 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:30.562921    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:22:30.740475 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:30.740440    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:30.799464 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:30.799430    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:40.562436 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:40.562392    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:22:50.590552 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:50.590512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:22:59.920610 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:59.920568    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:59.945326 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:59.945289    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"]
Aug 04 18:22:59.945540 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:59.945517    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kserve-container" containerID="cri-o://fae9e7e5ea8043e024cdd24a69f1b3abc764204831a167bf391da35c86747b26" gracePeriod=30
Aug 04 18:22:59.945615 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:22:59.945556    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kube-rbac-proxy" containerID="cri-o://0d5c4b003a5456440f1eba2327c89fa2b690609d6c14dc24a23746135ee8dc84" gracePeriod=30
Aug 04 18:23:00.557190 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:00.557143    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Aug 04 18:23:00.562723 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:00.562688    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 04 18:23:00.900682 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:00.900597    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerID="0d5c4b003a5456440f1eba2327c89fa2b690609d6c14dc24a23746135ee8dc84" exitCode=2
Aug 04 18:23:00.900682 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:00.900669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" event={"ID":"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a","Type":"ContainerDied","Data":"0d5c4b003a5456440f1eba2327c89fa2b690609d6c14dc24a23746135ee8dc84"}
Aug 04 18:23:04.921313 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:04.921273    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerID="fae9e7e5ea8043e024cdd24a69f1b3abc764204831a167bf391da35c86747b26" exitCode=0
Aug 04 18:23:04.921810 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:04.921343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" event={"ID":"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a","Type":"ContainerDied","Data":"fae9e7e5ea8043e024cdd24a69f1b3abc764204831a167bf391da35c86747b26"}
Aug 04 18:23:05.010848 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.010820    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:23:05.031697 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.031658    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:05.079659 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.079582    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"]
Aug 04 18:23:05.079887 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.079866    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kserve-container"
Aug 04 18:23:05.079887 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.079882    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kserve-container"
Aug 04 18:23:05.079887 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.079903    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kube-rbac-proxy"
Aug 04 18:23:05.079887 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.079911    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kube-rbac-proxy"
Aug 04 18:23:05.080244 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.079933    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="storage-initializer"
Aug 04 18:23:05.080244 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.079942    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="storage-initializer"
Aug 04 18:23:05.080244 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.080025    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kserve-container"
Aug 04 18:23:05.080244 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.080059    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" containerName="kube-rbac-proxy"
Aug 04 18:23:05.082768 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.082745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.085482 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.085460    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Aug 04 18:23:05.085482 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.085474    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Aug 04 18:23:05.094775 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.094729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"]
Aug 04 18:23:05.102119 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.102083    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kserve-provision-location\") pod \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") "
Aug 04 18:23:05.102332 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.102167    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-proxy-tls\") pod \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") "
Aug 04 18:23:05.102332 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.102208    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-cabundle-cert\") pod \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") "
Aug 04 18:23:05.102332 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.102236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg2hk\" (UniqueName: \"kubernetes.io/projected/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-api-access-hg2hk\") pod \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") "
Aug 04 18:23:05.102332 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.102285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-rbac-proxy-sar-config\") pod \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\" (UID: \"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a\") "
Aug 04 18:23:05.102547 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.102476    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" (UID: "6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:05.102606 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.102555    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" (UID: "6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:23:05.102795 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.102767    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" (UID: "6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:23:05.104754 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.104716    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" (UID: "6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:23:05.104884 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.104751    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-api-access-hg2hk" (OuterVolumeSpecName: "kube-api-access-hg2hk") pod "6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" (UID: "6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a"). InnerVolumeSpecName "kube-api-access-hg2hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:23:05.203316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.203277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.203316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.203320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszfk\" (UniqueName: \"kubernetes.io/projected/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-api-access-zszfk\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.203613 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.203338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.203613 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.203368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.203613 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.203466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.203613 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.203560    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:23:05.203613 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.203575    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg2hk\" (UniqueName: \"kubernetes.io/projected/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-api-access-hg2hk\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:23:05.203613 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.203585    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:23:05.203613 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.203594    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:23:05.203613 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.203603    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:23:05.215661 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.215628    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:05.304965 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.304916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.305190 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.305085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.305190 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.305138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszfk\" (UniqueName: \"kubernetes.io/projected/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-api-access-zszfk\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.305190 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.305165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.305361 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.305243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.305604 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.305568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.305716 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.305695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.305888 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.305869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.307807 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.307786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.316369 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.316343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszfk\" (UniqueName: \"kubernetes.io/projected/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-api-access-zszfk\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.393983 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.393890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:05.538566 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.538540    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"]
Aug 04 18:23:05.541017 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:23:05.540983    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2548fa2e_e193_4b20_a2d1_0d3f02ff8551.slice/crio-c924e92149fe548a0a6d157a6427c51fdfdd9167107394cdfc298320d908ec05 WatchSource:0}: Error finding container c924e92149fe548a0a6d157a6427c51fdfdd9167107394cdfc298320d908ec05: Status 404 returned error can't find the container with id c924e92149fe548a0a6d157a6427c51fdfdd9167107394cdfc298320d908ec05
Aug 04 18:23:05.927905 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.927860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" event={"ID":"2548fa2e-e193-4b20-a2d1-0d3f02ff8551","Type":"ContainerStarted","Data":"c86e757dcb4c9430602999d37afddd1660d9a78ee00ddf1a5954744061d11ae4"}
Aug 04 18:23:05.928501 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.927937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" event={"ID":"2548fa2e-e193-4b20-a2d1-0d3f02ff8551","Type":"ContainerStarted","Data":"c924e92149fe548a0a6d157a6427c51fdfdd9167107394cdfc298320d908ec05"}
Aug 04 18:23:05.930056 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.930011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9" event={"ID":"6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a","Type":"ContainerDied","Data":"f230084e9ce1574f03553f341a25d32a8e21da6104075ca201ceb81c5ef9f483"}
Aug 04 18:23:05.930170 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.930060    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"
Aug 04 18:23:05.930170 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.930100    2581 scope.go:117] "RemoveContainer" containerID="0d5c4b003a5456440f1eba2327c89fa2b690609d6c14dc24a23746135ee8dc84"
Aug 04 18:23:05.940595 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.940557    2581 scope.go:117] "RemoveContainer" containerID="fae9e7e5ea8043e024cdd24a69f1b3abc764204831a167bf391da35c86747b26"
Aug 04 18:23:05.949134 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.949111    2581 scope.go:117] "RemoveContainer" containerID="e464fffd201bd3faf996249a0ec26d1663f9c490fdef280df67d8f372aa78c65"
Aug 04 18:23:05.967551 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.967512    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"]
Aug 04 18:23:05.972104 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:05.972068    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-hs2j9"]
Aug 04 18:23:07.214950 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:07.214912    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a" path="/var/lib/kubelet/pods/6ddd38af-ef8e-4c30-b936-0d0bc2ddaa4a/volumes"
Aug 04 18:23:09.949616 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:09.949580    2581 generic.go:356] "Generic (PLEG): container finished" podID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerID="c86e757dcb4c9430602999d37afddd1660d9a78ee00ddf1a5954744061d11ae4" exitCode=0
Aug 04 18:23:09.950000 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:09.949655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" event={"ID":"2548fa2e-e193-4b20-a2d1-0d3f02ff8551","Type":"ContainerDied","Data":"c86e757dcb4c9430602999d37afddd1660d9a78ee00ddf1a5954744061d11ae4"}
Aug 04 18:23:10.956098 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:10.956059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" event={"ID":"2548fa2e-e193-4b20-a2d1-0d3f02ff8551","Type":"ContainerStarted","Data":"8fc58791fb2e57aae86bcc34bd69aaad00f46da05e67e391cd16b455517f1a7f"}
Aug 04 18:23:10.956561 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:10.956134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" event={"ID":"2548fa2e-e193-4b20-a2d1-0d3f02ff8551","Type":"ContainerStarted","Data":"781dd0da28229ceedb9c8dadd6aaef507269294cd2695aa41d2283c5c4849091"}
Aug 04 18:23:10.956561 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:10.956364    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:10.980712 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:10.980654    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" podStartSLOduration=2.354487964 podStartE2EDuration="5.980633529s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.626145565s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:23:10.978498983 +0000 UTC m=+1416.322900179" watchObservedRunningTime="2026-08-04 18:23:10.980633529 +0000 UTC m=+1416.325034749"
Aug 04 18:23:11.960236 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:11.960200    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:17.970585 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:17.970558    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:23:47.971512 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:47.971469    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:23:57.971338 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:23:57.971292    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:24:07.971731 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:07.971693    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:24:17.971964 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:17.971915    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:24:25.009504 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:25.009450    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:27.988625 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:27.988593    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:24:29.994891 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:29.994839    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:30.555286 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:30.555248    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:35.213890 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:35.213818    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:24:35.214834 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:35.214804    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:24:35.221856 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:35.221825    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:24:35.222426 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:35.222407    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:24:35.369649 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:35.369607    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"]
Aug 04 18:24:35.369971 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:35.369929    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kserve-container" containerID="cri-o://781dd0da28229ceedb9c8dadd6aaef507269294cd2695aa41d2283c5c4849091" gracePeriod=30
Aug 04 18:24:35.370098 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:35.369994    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kube-rbac-proxy" containerID="cri-o://8fc58791fb2e57aae86bcc34bd69aaad00f46da05e67e391cd16b455517f1a7f" gracePeriod=30
Aug 04 18:24:35.460793 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:35.460757    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:36.301091 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:36.301023    2581 generic.go:356] "Generic (PLEG): container finished" podID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerID="8fc58791fb2e57aae86bcc34bd69aaad00f46da05e67e391cd16b455517f1a7f" exitCode=2
Aug 04 18:24:36.301467 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:36.301097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" event={"ID":"2548fa2e-e193-4b20-a2d1-0d3f02ff8551","Type":"ContainerDied","Data":"8fc58791fb2e57aae86bcc34bd69aaad00f46da05e67e391cd16b455517f1a7f"}
Aug 04 18:24:37.964992 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:37.964943    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Aug 04 18:24:37.971429 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:37.971395    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.26:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:24:40.319037 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.318987    2581 generic.go:356] "Generic (PLEG): container finished" podID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerID="781dd0da28229ceedb9c8dadd6aaef507269294cd2695aa41d2283c5c4849091" exitCode=0
Aug 04 18:24:40.319467 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.319080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" event={"ID":"2548fa2e-e193-4b20-a2d1-0d3f02ff8551","Type":"ContainerDied","Data":"781dd0da28229ceedb9c8dadd6aaef507269294cd2695aa41d2283c5c4849091"}
Aug 04 18:24:40.413725 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.413696    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:24:40.506726 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.506631    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-cabundle-cert\") pod \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") "
Aug 04 18:24:40.506726 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.506685    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kserve-provision-location\") pod \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") "
Aug 04 18:24:40.506726 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.506710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-rbac-proxy-sar-config\") pod \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") "
Aug 04 18:24:40.507011 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.506741    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zszfk\" (UniqueName: \"kubernetes.io/projected/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-api-access-zszfk\") pod \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") "
Aug 04 18:24:40.507011 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.506775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-proxy-tls\") pod \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\" (UID: \"2548fa2e-e193-4b20-a2d1-0d3f02ff8551\") "
Aug 04 18:24:40.507141 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.507081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2548fa2e-e193-4b20-a2d1-0d3f02ff8551" (UID: "2548fa2e-e193-4b20-a2d1-0d3f02ff8551"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:24:40.507141 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.507101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2548fa2e-e193-4b20-a2d1-0d3f02ff8551" (UID: "2548fa2e-e193-4b20-a2d1-0d3f02ff8551"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:24:40.507141 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.507127    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2548fa2e-e193-4b20-a2d1-0d3f02ff8551" (UID: "2548fa2e-e193-4b20-a2d1-0d3f02ff8551"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:24:40.509110 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.509077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2548fa2e-e193-4b20-a2d1-0d3f02ff8551" (UID: "2548fa2e-e193-4b20-a2d1-0d3f02ff8551"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:24:40.509219 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.509147    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-api-access-zszfk" (OuterVolumeSpecName: "kube-api-access-zszfk") pod "2548fa2e-e193-4b20-a2d1-0d3f02ff8551" (UID: "2548fa2e-e193-4b20-a2d1-0d3f02ff8551"). InnerVolumeSpecName "kube-api-access-zszfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:24:40.591069 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.591011    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"]
Aug 04 18:24:40.591346 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.591333    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kube-rbac-proxy"
Aug 04 18:24:40.591388 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.591347    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kube-rbac-proxy"
Aug 04 18:24:40.591421 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.591368    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kserve-container"
Aug 04 18:24:40.591421 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.591399    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kserve-container"
Aug 04 18:24:40.591421 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.591405    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="storage-initializer"
Aug 04 18:24:40.591421 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.591410    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="storage-initializer"
Aug 04 18:24:40.591542 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.591455    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kserve-container"
Aug 04 18:24:40.591542 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.591467    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" containerName="kube-rbac-proxy"
Aug 04 18:24:40.593731 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.593714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.597658 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.597635    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Aug 04 18:24:40.598480 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.598463    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Aug 04 18:24:40.607823 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.607790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.608019 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.607855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.608019 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.607921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jzz\" (UniqueName: \"kubernetes.io/projected/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-api-access-n6jzz\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.608019 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.607949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.608019 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.607968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.608219 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.608057    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:24:40.608219 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.608074    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:24:40.608219 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.608088    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:24:40.608219 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.608099    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:24:40.608219 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.608108    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zszfk\" (UniqueName: \"kubernetes.io/projected/2548fa2e-e193-4b20-a2d1-0d3f02ff8551-kube-api-access-zszfk\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:24:40.611327 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.611303    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"]
Aug 04 18:24:40.709334 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.709290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.709529 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.709356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.709529 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.709380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jzz\" (UniqueName: \"kubernetes.io/projected/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-api-access-n6jzz\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.709529 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.709399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.709529 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.709419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.709931 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.709898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.710133 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.710115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.710255 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.710238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.711945 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.711928    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.729898 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.729867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jzz\" (UniqueName: \"kubernetes.io/projected/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-api-access-n6jzz\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:40.903743 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:40.903704    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:41.054741 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.054707    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"]
Aug 04 18:24:41.058227 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:24:41.058198    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41a35b4_45da_4ffe_ba82_746a8c5c8f80.slice/crio-1e3e68c52ccacbe923bde67196bf8d9e04e9de740704198ba0344f0f25a20134 WatchSource:0}: Error finding container 1e3e68c52ccacbe923bde67196bf8d9e04e9de740704198ba0344f0f25a20134: Status 404 returned error can't find the container with id 1e3e68c52ccacbe923bde67196bf8d9e04e9de740704198ba0344f0f25a20134
Aug 04 18:24:41.243705 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.243674    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:41.324631 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.324596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr" event={"ID":"2548fa2e-e193-4b20-a2d1-0d3f02ff8551","Type":"ContainerDied","Data":"c924e92149fe548a0a6d157a6427c51fdfdd9167107394cdfc298320d908ec05"}
Aug 04 18:24:41.325103 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.324668    2581 scope.go:117] "RemoveContainer" containerID="8fc58791fb2e57aae86bcc34bd69aaad00f46da05e67e391cd16b455517f1a7f"
Aug 04 18:24:41.325103 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.324606    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"
Aug 04 18:24:41.326552 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.326525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" event={"ID":"e41a35b4-45da-4ffe-ba82-746a8c5c8f80","Type":"ContainerStarted","Data":"0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8"}
Aug 04 18:24:41.326685 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.326576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" event={"ID":"e41a35b4-45da-4ffe-ba82-746a8c5c8f80","Type":"ContainerStarted","Data":"1e3e68c52ccacbe923bde67196bf8d9e04e9de740704198ba0344f0f25a20134"}
Aug 04 18:24:41.333098 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.333076    2581 scope.go:117] "RemoveContainer" containerID="781dd0da28229ceedb9c8dadd6aaef507269294cd2695aa41d2283c5c4849091"
Aug 04 18:24:41.341522 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.341496    2581 scope.go:117] "RemoveContainer" containerID="c86e757dcb4c9430602999d37afddd1660d9a78ee00ddf1a5954744061d11ae4"
Aug 04 18:24:41.351315 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.351192    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"]
Aug 04 18:24:41.353298 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.353269    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-mjxwr"]
Aug 04 18:24:41.362942 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:41.362897    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:43.214561 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:43.214521    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2548fa2e-e193-4b20-a2d1-0d3f02ff8551" path="/var/lib/kubelet/pods/2548fa2e-e193-4b20-a2d1-0d3f02ff8551/volumes"
Aug 04 18:24:45.344216 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:45.344183    2581 generic.go:356] "Generic (PLEG): container finished" podID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerID="0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8" exitCode=0
Aug 04 18:24:45.344598 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:45.344253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" event={"ID":"e41a35b4-45da-4ffe-ba82-746a8c5c8f80","Type":"ContainerDied","Data":"0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8"}
Aug 04 18:24:46.350501 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:46.350464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" event={"ID":"e41a35b4-45da-4ffe-ba82-746a8c5c8f80","Type":"ContainerStarted","Data":"ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24"}
Aug 04 18:24:46.350873 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:46.350527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" event={"ID":"e41a35b4-45da-4ffe-ba82-746a8c5c8f80","Type":"ContainerStarted","Data":"bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035"}
Aug 04 18:24:46.350873 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:46.350739    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:47.354639 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:47.354608    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:53.364994 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:53.364958    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:24:53.394449 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:24:53.394392    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" podStartSLOduration=9.654684963 podStartE2EDuration="13.394375052s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.739690089s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:24:46.386244166 +0000 UTC m=+1511.730645350" watchObservedRunningTime="2026-08-04 18:24:53.394375052 +0000 UTC m=+1518.738776208"
Aug 04 18:25:23.366777 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:25:23.366735    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:25:33.365853 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:25:33.365813    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:25:43.366160 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:25:43.366117    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:25:53.365722 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:25:53.365671    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:26:03.388522 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:03.388484    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:26:10.624199 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:10.624157    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"]
Aug 04 18:26:10.624769 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:10.624471    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kserve-container" containerID="cri-o://bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035" gracePeriod=30
Aug 04 18:26:10.624769 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:10.624510    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kube-rbac-proxy" containerID="cri-o://ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24" gracePeriod=30
Aug 04 18:26:10.639172 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:10.639148    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:11.690780 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:11.690744    2581 generic.go:356] "Generic (PLEG): container finished" podID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerID="ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24" exitCode=2
Aug 04 18:26:11.691198 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:11.690820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" event={"ID":"e41a35b4-45da-4ffe-ba82-746a8c5c8f80","Type":"ContainerDied","Data":"ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24"}
Aug 04 18:26:13.359932 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:13.359887    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Aug 04 18:26:13.366460 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:13.366424    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 04 18:26:15.824392 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:15.824353    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:15.948712 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:15.948666    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:16.367004 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.366980    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:26:16.488387 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.488285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-cabundle-cert\") pod \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") "
Aug 04 18:26:16.488387 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.488326    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-rbac-proxy-sar-config\") pod \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") "
Aug 04 18:26:16.488387 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.488382    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-proxy-tls\") pod \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") "
Aug 04 18:26:16.488705 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.488408    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kserve-provision-location\") pod \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") "
Aug 04 18:26:16.488705 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.488431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6jzz\" (UniqueName: \"kubernetes.io/projected/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-api-access-n6jzz\") pod \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\" (UID: \"e41a35b4-45da-4ffe-ba82-746a8c5c8f80\") "
Aug 04 18:26:16.488817 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.488780    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e41a35b4-45da-4ffe-ba82-746a8c5c8f80" (UID: "e41a35b4-45da-4ffe-ba82-746a8c5c8f80"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:26:16.488817 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.488780    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e41a35b4-45da-4ffe-ba82-746a8c5c8f80" (UID: "e41a35b4-45da-4ffe-ba82-746a8c5c8f80"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:16.488817 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.488788    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e41a35b4-45da-4ffe-ba82-746a8c5c8f80" (UID: "e41a35b4-45da-4ffe-ba82-746a8c5c8f80"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:26:16.490725 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.490691    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e41a35b4-45da-4ffe-ba82-746a8c5c8f80" (UID: "e41a35b4-45da-4ffe-ba82-746a8c5c8f80"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:26:16.490823 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.490800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-api-access-n6jzz" (OuterVolumeSpecName: "kube-api-access-n6jzz") pod "e41a35b4-45da-4ffe-ba82-746a8c5c8f80" (UID: "e41a35b4-45da-4ffe-ba82-746a8c5c8f80"). InnerVolumeSpecName "kube-api-access-n6jzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:26:16.512682 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.512648    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:16.589343 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.589294    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:26:16.589343 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.589335    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:26:16.589343 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.589346    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6jzz\" (UniqueName: \"kubernetes.io/projected/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-api-access-n6jzz\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:26:16.589343 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.589355    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:26:16.589614 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.589364    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e41a35b4-45da-4ffe-ba82-746a8c5c8f80-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:26:16.711034 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.710999    2581 generic.go:356] "Generic (PLEG): container finished" podID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerID="bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035" exitCode=0
Aug 04 18:26:16.711259 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.711076    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"
Aug 04 18:26:16.711259 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.711083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" event={"ID":"e41a35b4-45da-4ffe-ba82-746a8c5c8f80","Type":"ContainerDied","Data":"bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035"}
Aug 04 18:26:16.711259 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.711141    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m" event={"ID":"e41a35b4-45da-4ffe-ba82-746a8c5c8f80","Type":"ContainerDied","Data":"1e3e68c52ccacbe923bde67196bf8d9e04e9de740704198ba0344f0f25a20134"}
Aug 04 18:26:16.711259 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.711170    2581 scope.go:117] "RemoveContainer" containerID="ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24"
Aug 04 18:26:16.720413 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.720396    2581 scope.go:117] "RemoveContainer" containerID="bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035"
Aug 04 18:26:16.728608 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.728586    2581 scope.go:117] "RemoveContainer" containerID="0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8"
Aug 04 18:26:16.735991 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.735960    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"]
Aug 04 18:26:16.737911 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.737885    2581 scope.go:117] "RemoveContainer" containerID="ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24"
Aug 04 18:26:16.738300 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:26:16.738267    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24\": container with ID starting with ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24 not found: ID does not exist" containerID="ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24"
Aug 04 18:26:16.738372 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.738311    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24"} err="failed to get container status \"ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24\": rpc error: code = NotFound desc = could not find container \"ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24\": container with ID starting with ac84e67bfe3e6f38a8becf90695c7e9e67d283c7b8a149d6bf965ba7cc6a5d24 not found: ID does not exist"
Aug 04 18:26:16.738372 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.738334    2581 scope.go:117] "RemoveContainer" containerID="bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035"
Aug 04 18:26:16.738662 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:26:16.738614    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035\": container with ID starting with bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035 not found: ID does not exist" containerID="bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035"
Aug 04 18:26:16.738662 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.738641    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035"} err="failed to get container status \"bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035\": rpc error: code = NotFound desc = could not find container \"bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035\": container with ID starting with bb115c1fb0a37011bfa7a6db24c6cdafe0cee09a6e7af5dde55fb502684fd035 not found: ID does not exist"
Aug 04 18:26:16.738784 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.738664    2581 scope.go:117] "RemoveContainer" containerID="0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8"
Aug 04 18:26:16.738924 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:26:16.738907    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8\": container with ID starting with 0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8 not found: ID does not exist" containerID="0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8"
Aug 04 18:26:16.738964 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.738928    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8"} err="failed to get container status \"0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8\": rpc error: code = NotFound desc = could not find container \"0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8\": container with ID starting with 0ee50fe049db51b9b24808c48826c43b6a22089bd93b216e70473b5b46c5c9a8 not found: ID does not exist"
Aug 04 18:26:16.741394 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:16.741370    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fjm7m"]
Aug 04 18:26:17.215264 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:17.215217    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" path="/var/lib/kubelet/pods/e41a35b4-45da-4ffe-ba82-746a8c5c8f80/volumes"
Aug 04 18:26:17.913570 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:17.913540    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:21.573277 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:21.573236    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:21.831595 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:21.831506    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:22.989010 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:22.988973    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:23.233460 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:26:23.233426    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:04.258098 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:04.258062    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:09.338640 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.338600    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:09.376477 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.376443    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"]
Aug 04 18:27:09.376746 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.376733    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="storage-initializer"
Aug 04 18:27:09.376805 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.376747    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="storage-initializer"
Aug 04 18:27:09.376805 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.376760    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kserve-container"
Aug 04 18:27:09.376805 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.376766    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kserve-container"
Aug 04 18:27:09.376805 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.376779    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kube-rbac-proxy"
Aug 04 18:27:09.376805 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.376785    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kube-rbac-proxy"
Aug 04 18:27:09.376969 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.376834    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kube-rbac-proxy"
Aug 04 18:27:09.376969 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.376842    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e41a35b4-45da-4ffe-ba82-746a8c5c8f80" containerName="kserve-container"
Aug 04 18:27:09.379208 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.379189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.381599 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.381579    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:27:09.381966 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.381947    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:27:09.381966 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.381959    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:27:09.382231 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.382217    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:27:09.382402 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.382386    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Aug 04 18:27:09.382473 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.382412    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Aug 04 18:27:09.382473 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.382436    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:27:09.390232 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.390192    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"]
Aug 04 18:27:09.409686 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.409636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwltq\" (UniqueName: \"kubernetes.io/projected/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-api-access-bwltq\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.409935 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.409906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.410114 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.410076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.410246 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.410122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.410246 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.410149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.510870 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.510826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.510870 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.510875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.511137 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:27:09.510976    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Aug 04 18:27:09.511137 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.510988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.511137 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.511021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.511137 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:27:09.511071    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-proxy-tls podName:d0d4f635-5d18-4cbe-9773-3491b3d3f8a4 nodeName:}" failed. No retries permitted until 2026-08-04 18:27:10.011024476 +0000 UTC m=+1655.355425611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" (UID: "d0d4f635-5d18-4cbe-9773-3491b3d3f8a4") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Aug 04 18:27:09.511137 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.511121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwltq\" (UniqueName: \"kubernetes.io/projected/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-api-access-bwltq\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.511406 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.511284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.511683 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.511664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.511721 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.511679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.524939 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.524910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwltq\" (UniqueName: \"kubernetes.io/projected/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-api-access-bwltq\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:09.746816 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:09.746742    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:10.015205 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:10.015112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:10.017742 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:10.017708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:10.290642 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:10.290607    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:27:10.431394 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:10.431365    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"]
Aug 04 18:27:10.434012 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:27:10.433989    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d4f635_5d18_4cbe_9773_3491b3d3f8a4.slice/crio-adf99dcab5cf2af3f29ffcbf16a669f83e5affdb37c0374c8ff2884cf44351ca WatchSource:0}: Error finding container adf99dcab5cf2af3f29ffcbf16a669f83e5affdb37c0374c8ff2884cf44351ca: Status 404 returned error can't find the container with id adf99dcab5cf2af3f29ffcbf16a669f83e5affdb37c0374c8ff2884cf44351ca
Aug 04 18:27:10.927082 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:10.927019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" event={"ID":"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4","Type":"ContainerStarted","Data":"008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91"}
Aug 04 18:27:10.927287 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:10.927134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" event={"ID":"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4","Type":"ContainerStarted","Data":"adf99dcab5cf2af3f29ffcbf16a669f83e5affdb37c0374c8ff2884cf44351ca"}
Aug 04 18:27:19.967530 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:19.967494    2581 generic.go:356] "Generic (PLEG): container finished" podID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerID="008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91" exitCode=0
Aug 04 18:27:19.967906 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:19.967545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" event={"ID":"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4","Type":"ContainerDied","Data":"008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91"}
Aug 04 18:27:19.968857 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:19.968839    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:27:53.250033 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:53.249992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:58.274094 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:58.273981    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:58.729884 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:27:58.729803    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:12.752562 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:28:12.752524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:28:12.822436 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:28:12.822391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:27.539111 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:27.539075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" event={"ID":"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4","Type":"ContainerStarted","Data":"fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369"}
Aug 04 18:29:28.546966 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:28.546926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" event={"ID":"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4","Type":"ContainerStarted","Data":"a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e"}
Aug 04 18:29:28.547567 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:28.547161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:29:28.577073 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:28.577002    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" podStartSLOduration=3.503634104 podStartE2EDuration="2m19.576983697s" totalImagesPullingTime="2m7.365982246s" totalInitContainerRuntime="8.707367347s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:29:28.574768972 +0000 UTC m=+1793.919170150" watchObservedRunningTime="2026-08-04 18:29:28.576983697 +0000 UTC m=+1793.921384855"
Aug 04 18:29:29.551890 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:29.551855    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:29:29.553313 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:29.553280    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:29:30.555857 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:30.555811    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:29:35.244395 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:35.244366    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:29:35.244871 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:35.244420    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:29:35.253232 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:35.253203    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:29:35.253417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:35.253362    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:29:35.561159 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:35.561125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:29:35.561719 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:35.561684    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:29:37.849932 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:37.849897    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:45.562440 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:45.562402    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:29:54.948655 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:54.948601    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"]
Aug 04 18:29:54.949108 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:54.948947    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kserve-container" containerID="cri-o://fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369" gracePeriod=30
Aug 04 18:29:54.949108 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:54.949007    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kube-rbac-proxy" containerID="cri-o://a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e" gracePeriod=30
Aug 04 18:29:54.951568 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:54.951534    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:29:55.556728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:55.556683    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Aug 04 18:29:55.562215 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:55.562176    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:29:55.660643 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:55.660608    2581 generic.go:356] "Generic (PLEG): container finished" podID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerID="a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e" exitCode=2
Aug 04 18:29:55.660816 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:55.660655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" event={"ID":"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4","Type":"ContainerDied","Data":"a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e"}
Aug 04 18:29:56.513429 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.513388    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:29:56.587943 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.587847    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kserve-provision-location\") pod \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") "
Aug 04 18:29:56.587943 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.587885    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-rbac-proxy-sar-config\") pod \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") "
Aug 04 18:29:56.587943 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.587916    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-cabundle-cert\") pod \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") "
Aug 04 18:29:56.588260 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.587956    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-proxy-tls\") pod \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") "
Aug 04 18:29:56.588260 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.587979    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwltq\" (UniqueName: \"kubernetes.io/projected/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-api-access-bwltq\") pod \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\" (UID: \"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4\") "
Aug 04 18:29:56.588373 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.588341    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" (UID: "d0d4f635-5d18-4cbe-9773-3491b3d3f8a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:29:56.588373 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.588350    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" (UID: "d0d4f635-5d18-4cbe-9773-3491b3d3f8a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:29:56.588959 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.588933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" (UID: "d0d4f635-5d18-4cbe-9773-3491b3d3f8a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:56.590437 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.590406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" (UID: "d0d4f635-5d18-4cbe-9773-3491b3d3f8a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:29:56.590437 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.590413    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-api-access-bwltq" (OuterVolumeSpecName: "kube-api-access-bwltq") pod "d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" (UID: "d0d4f635-5d18-4cbe-9773-3491b3d3f8a4"). InnerVolumeSpecName "kube-api-access-bwltq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:29:56.666739 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.666692    2581 generic.go:356] "Generic (PLEG): container finished" podID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerID="fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369" exitCode=0
Aug 04 18:29:56.666906 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.666750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" event={"ID":"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4","Type":"ContainerDied","Data":"fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369"}
Aug 04 18:29:56.666906 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.666764    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"
Aug 04 18:29:56.666906 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.666816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww" event={"ID":"d0d4f635-5d18-4cbe-9773-3491b3d3f8a4","Type":"ContainerDied","Data":"adf99dcab5cf2af3f29ffcbf16a669f83e5affdb37c0374c8ff2884cf44351ca"}
Aug 04 18:29:56.666906 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.666854    2581 scope.go:117] "RemoveContainer" containerID="a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e"
Aug 04 18:29:56.676203 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.676180    2581 scope.go:117] "RemoveContainer" containerID="fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369"
Aug 04 18:29:56.684472 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.684453    2581 scope.go:117] "RemoveContainer" containerID="008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91"
Aug 04 18:29:56.688799 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.688776    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:29:56.688912 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.688801    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:29:56.688912 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.688819    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:29:56.688912 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.688832    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:29:56.688912 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.688875    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwltq\" (UniqueName: \"kubernetes.io/projected/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4-kube-api-access-bwltq\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:29:56.692687 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.692663    2581 scope.go:117] "RemoveContainer" containerID="a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e"
Aug 04 18:29:56.693034 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:29:56.693014    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e\": container with ID starting with a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e not found: ID does not exist" containerID="a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e"
Aug 04 18:29:56.693135 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.693085    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e"} err="failed to get container status \"a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e\": rpc error: code = NotFound desc = could not find container \"a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e\": container with ID starting with a4dff5162351c4c81b1e5f084ac2f22eddadeeb17c64135dc543caae23f7d88e not found: ID does not exist"
Aug 04 18:29:56.693135 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.693113    2581 scope.go:117] "RemoveContainer" containerID="fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369"
Aug 04 18:29:56.693436 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:29:56.693412    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369\": container with ID starting with fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369 not found: ID does not exist" containerID="fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369"
Aug 04 18:29:56.693482 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.693443    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369"} err="failed to get container status \"fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369\": rpc error: code = NotFound desc = could not find container \"fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369\": container with ID starting with fe14d066cb97144b0bc24a1e5859b56ad71a4c6b444fba1bb6bbf621cb69e369 not found: ID does not exist"
Aug 04 18:29:56.693482 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.693460    2581 scope.go:117] "RemoveContainer" containerID="008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91"
Aug 04 18:29:56.693681 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:29:56.693659    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91\": container with ID starting with 008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91 not found: ID does not exist" containerID="008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91"
Aug 04 18:29:56.693737 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.693689    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91"} err="failed to get container status \"008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91\": rpc error: code = NotFound desc = could not find container \"008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91\": container with ID starting with 008957bfde5cb41ae36da931e860bd35b8c1f9eb104b7f3861bfffc01e468a91 not found: ID does not exist"
Aug 04 18:29:56.696806 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.696780    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"]
Aug 04 18:29:56.704134 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:56.704107    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vz9ww"]
Aug 04 18:29:57.214495 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:29:57.214458    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" path="/var/lib/kubelet/pods/d0d4f635-5d18-4cbe-9773-3491b3d3f8a4/volumes"
Aug 04 18:30:00.105426 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:30:00.105391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:00.167746 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:30:00.167710    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:00.281918 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:30:00.281876    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:01.030633 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:30:01.030603    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:39.107785 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:30:39.107748    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:39.168474 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:30:39.168443    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:58.557239 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:30:58.557194    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:03.624273 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:31:03.624238    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:03.990530 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:31:03.990445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:36.099577 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:31:36.099542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:41.185905 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:31:41.185859    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:41.815255 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:31:41.815213    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:50.179641 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:31:50.179604    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:50.297235 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:31:50.297196    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:55.304097 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:31:55.304061    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:55.367475 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:31:55.367434    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:55.565466 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:31:55.565383    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:31:55.629330 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:31:55.629294    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:41.337315 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:32:41.337231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:46.478755 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:32:46.478711    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:46.715029 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:32:46.714992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:25.499590 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.499551    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:25.516082 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.516030    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"]
Aug 04 18:33:25.516372 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.516359    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kube-rbac-proxy"
Aug 04 18:33:25.516435 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.516374    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kube-rbac-proxy"
Aug 04 18:33:25.516435 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.516382    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kserve-container"
Aug 04 18:33:25.516435 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.516388    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kserve-container"
Aug 04 18:33:25.516435 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.516405    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="storage-initializer"
Aug 04 18:33:25.516435 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.516411    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="storage-initializer"
Aug 04 18:33:25.516627 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.516463    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kube-rbac-proxy"
Aug 04 18:33:25.516627 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.516472    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d4f635-5d18-4cbe-9773-3491b3d3f8a4" containerName="kserve-container"
Aug 04 18:33:25.518971 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.518951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.521386 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.521354    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:33:25.521539 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.521438    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Aug 04 18:33:25.521539 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.521451    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:33:25.521539 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.521495    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:33:25.522206 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.522187    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Aug 04 18:33:25.522318 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.522209    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:33:25.522318 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.522213    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:33:25.533401 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.533367    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"]
Aug 04 18:33:25.658695 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.658658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.658883 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.658709    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/931c40f8-d384-454c-858f-9efa49d5b736-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.658883 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.658743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/931c40f8-d384-454c-858f-9efa49d5b736-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.658883 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.658767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.658883 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.658804    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f84n6\" (UniqueName: \"kubernetes.io/projected/931c40f8-d384-454c-858f-9efa49d5b736-kube-api-access-f84n6\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.759451 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.759362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.759451 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.759410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/931c40f8-d384-454c-858f-9efa49d5b736-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.759451 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.759432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/931c40f8-d384-454c-858f-9efa49d5b736-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.759451 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.759455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.759794 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.759478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f84n6\" (UniqueName: \"kubernetes.io/projected/931c40f8-d384-454c-858f-9efa49d5b736-kube-api-access-f84n6\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.759893 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.759866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/931c40f8-d384-454c-858f-9efa49d5b736-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.760195 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.760174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.760249 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.760218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.762190 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.762169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/931c40f8-d384-454c-858f-9efa49d5b736-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.770710 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.770658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f84n6\" (UniqueName: \"kubernetes.io/projected/931c40f8-d384-454c-858f-9efa49d5b736-kube-api-access-f84n6\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.830338 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.830294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:25.977826 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:25.977749    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"]
Aug 04 18:33:25.980706 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:33:25.980677    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931c40f8_d384_454c_858f_9efa49d5b736.slice/crio-77b7c4d46e9290c3ccc592535e2a84e093315a360dde6b318af712c9bbfa88a8 WatchSource:0}: Error finding container 77b7c4d46e9290c3ccc592535e2a84e093315a360dde6b318af712c9bbfa88a8: Status 404 returned error can't find the container with id 77b7c4d46e9290c3ccc592535e2a84e093315a360dde6b318af712c9bbfa88a8
Aug 04 18:33:26.481477 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:26.481432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" event={"ID":"931c40f8-d384-454c-858f-9efa49d5b736","Type":"ContainerStarted","Data":"966c3a70302296b74857937af517f3fee23ede2241727f2bcd1f2017182d1b30"}
Aug 04 18:33:26.481657 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:26.481508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" event={"ID":"931c40f8-d384-454c-858f-9efa49d5b736","Type":"ContainerStarted","Data":"77b7c4d46e9290c3ccc592535e2a84e093315a360dde6b318af712c9bbfa88a8"}
Aug 04 18:33:30.499368 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:30.499337    2581 generic.go:356] "Generic (PLEG): container finished" podID="931c40f8-d384-454c-858f-9efa49d5b736" containerID="966c3a70302296b74857937af517f3fee23ede2241727f2bcd1f2017182d1b30" exitCode=0
Aug 04 18:33:30.499768 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:30.499391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" event={"ID":"931c40f8-d384-454c-858f-9efa49d5b736","Type":"ContainerDied","Data":"966c3a70302296b74857937af517f3fee23ede2241727f2bcd1f2017182d1b30"}
Aug 04 18:33:30.500522 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:30.500503    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:33:44.581970 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:44.581929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" event={"ID":"931c40f8-d384-454c-858f-9efa49d5b736","Type":"ContainerStarted","Data":"2bdb78afd61e923ed3420855aa5203316c3a0bd2a3532291ba28b5260e590cd8"}
Aug 04 18:33:44.582399 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:44.581997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" event={"ID":"931c40f8-d384-454c-858f-9efa49d5b736","Type":"ContainerStarted","Data":"f80beb104318c092e0c711c73db9deec428d6e7d3408d263f7dbf440de2ed520"}
Aug 04 18:33:44.582399 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:44.582257    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:45.586085 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:45.586052    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:45.587401 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:45.587363    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:33:46.589660 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:46.589625    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:33:51.596826 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:51.596794    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:33:51.597331 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:51.597301    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:33:51.630367 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:33:51.630295    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podStartSLOduration=9.092611822 podStartE2EDuration="26.630264755s" totalImagesPullingTime="13.721483994s" totalInitContainerRuntime="3.816168939s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:33:44.608604959 +0000 UTC m=+2049.953006116" watchObservedRunningTime="2026-08-04 18:33:51.630264755 +0000 UTC m=+2056.974665912"
Aug 04 18:34:01.597491 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:01.597449    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:34:11.597935 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:11.597886    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:34:16.855908 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:16.855876    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:21.597718 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:21.597679    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:34:22.056712 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:22.056673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:22.300177 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:22.300139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:31.597678 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:31.597623    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:34:35.270932 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:35.270896    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:34:35.271743 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:35.271722    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:34:35.277936 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:35.277912    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:34:35.278444 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:35.278428    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:34:41.597544 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:41.597500    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:34:51.597677 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:34:51.597638    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:35:01.215449 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:01.215420    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:35:05.758384 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:05.758349    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"]
Aug 04 18:35:05.758801 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:05.758614    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container" containerID="cri-o://f80beb104318c092e0c711c73db9deec428d6e7d3408d263f7dbf440de2ed520" gracePeriod=30
Aug 04 18:35:05.758801 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:05.758632    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kube-rbac-proxy" containerID="cri-o://2bdb78afd61e923ed3420855aa5203316c3a0bd2a3532291ba28b5260e590cd8" gracePeriod=30
Aug 04 18:35:05.797435 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:05.797400    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:05.904747 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:05.904714    2581 generic.go:356] "Generic (PLEG): container finished" podID="931c40f8-d384-454c-858f-9efa49d5b736" containerID="2bdb78afd61e923ed3420855aa5203316c3a0bd2a3532291ba28b5260e590cd8" exitCode=2
Aug 04 18:35:05.904950 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:05.904769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" event={"ID":"931c40f8-d384-454c-858f-9efa49d5b736","Type":"ContainerDied","Data":"2bdb78afd61e923ed3420855aa5203316c3a0bd2a3532291ba28b5260e590cd8"}
Aug 04 18:35:06.589978 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:06.589923    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Aug 04 18:35:10.928060 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:10.928001    2581 generic.go:356] "Generic (PLEG): container finished" podID="931c40f8-d384-454c-858f-9efa49d5b736" containerID="f80beb104318c092e0c711c73db9deec428d6e7d3408d263f7dbf440de2ed520" exitCode=0
Aug 04 18:35:10.928519 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:10.928077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" event={"ID":"931c40f8-d384-454c-858f-9efa49d5b736","Type":"ContainerDied","Data":"f80beb104318c092e0c711c73db9deec428d6e7d3408d263f7dbf440de2ed520"}
Aug 04 18:35:10.938673 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:10.938624    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:11.019287 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.019258    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:35:11.040808 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.040711    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:11.113031 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.112992    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/931c40f8-d384-454c-858f-9efa49d5b736-kserve-provision-location\") pod \"931c40f8-d384-454c-858f-9efa49d5b736\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") "
Aug 04 18:35:11.113237 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.113085    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/931c40f8-d384-454c-858f-9efa49d5b736-proxy-tls\") pod \"931c40f8-d384-454c-858f-9efa49d5b736\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") "
Aug 04 18:35:11.113237 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.113123    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f84n6\" (UniqueName: \"kubernetes.io/projected/931c40f8-d384-454c-858f-9efa49d5b736-kube-api-access-f84n6\") pod \"931c40f8-d384-454c-858f-9efa49d5b736\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") "
Aug 04 18:35:11.113237 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.113151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-kube-rbac-proxy-sar-config\") pod \"931c40f8-d384-454c-858f-9efa49d5b736\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") "
Aug 04 18:35:11.113237 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.113181    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-cabundle-cert\") pod \"931c40f8-d384-454c-858f-9efa49d5b736\" (UID: \"931c40f8-d384-454c-858f-9efa49d5b736\") "
Aug 04 18:35:11.113466 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.113438    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931c40f8-d384-454c-858f-9efa49d5b736-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "931c40f8-d384-454c-858f-9efa49d5b736" (UID: "931c40f8-d384-454c-858f-9efa49d5b736"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:35:11.113528 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.113511    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "931c40f8-d384-454c-858f-9efa49d5b736" (UID: "931c40f8-d384-454c-858f-9efa49d5b736"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:35:11.113576 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.113555    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "931c40f8-d384-454c-858f-9efa49d5b736" (UID: "931c40f8-d384-454c-858f-9efa49d5b736"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:35:11.115543 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.115511    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931c40f8-d384-454c-858f-9efa49d5b736-kube-api-access-f84n6" (OuterVolumeSpecName: "kube-api-access-f84n6") pod "931c40f8-d384-454c-858f-9efa49d5b736" (UID: "931c40f8-d384-454c-858f-9efa49d5b736"). InnerVolumeSpecName "kube-api-access-f84n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:35:11.115695 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.115603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/931c40f8-d384-454c-858f-9efa49d5b736-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "931c40f8-d384-454c-858f-9efa49d5b736" (UID: "931c40f8-d384-454c-858f-9efa49d5b736"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:35:11.163319 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.163282    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"]
Aug 04 18:35:11.163575 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.163563    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="storage-initializer"
Aug 04 18:35:11.163663 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.163577    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="storage-initializer"
Aug 04 18:35:11.163663 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.163599    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container"
Aug 04 18:35:11.163663 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.163605    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container"
Aug 04 18:35:11.163663 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.163614    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kube-rbac-proxy"
Aug 04 18:35:11.163663 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.163619    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kube-rbac-proxy"
Aug 04 18:35:11.163663 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.163661    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kube-rbac-proxy"
Aug 04 18:35:11.163835 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.163668    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="931c40f8-d384-454c-858f-9efa49d5b736" containerName="kserve-container"
Aug 04 18:35:11.166227 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.166202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.174872 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.174849    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Aug 04 18:35:11.175128 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.175113    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Aug 04 18:35:11.187248 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.187214    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"]
Aug 04 18:35:11.213766 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.213731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7sc\" (UniqueName: \"kubernetes.io/projected/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-api-access-qp7sc\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.213766 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.213776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.214113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.213825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.214113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.213848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779885fa-3b76-4d94-9957-dca6b1b0d1c5-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.214113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.213867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.214113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.213944    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/931c40f8-d384-454c-858f-9efa49d5b736-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:35:11.214113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.213960    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/931c40f8-d384-454c-858f-9efa49d5b736-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:35:11.214113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.213975    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f84n6\" (UniqueName: \"kubernetes.io/projected/931c40f8-d384-454c-858f-9efa49d5b736-kube-api-access-f84n6\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:35:11.214113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.213986    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:35:11.214113 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.213995    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/931c40f8-d384-454c-858f-9efa49d5b736-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:35:11.215094 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.215060    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:11.315349 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.315252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7sc\" (UniqueName: \"kubernetes.io/projected/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-api-access-qp7sc\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.315349 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.315297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.315349 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.315334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.315349 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.315353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779885fa-3b76-4d94-9957-dca6b1b0d1c5-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.315679 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.315375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.315925 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.315882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.316122 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.316102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.316199 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.316106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.318212 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.318192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779885fa-3b76-4d94-9957-dca6b1b0d1c5-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.329208 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.329175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7sc\" (UniqueName: \"kubernetes.io/projected/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-api-access-qp7sc\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-b82lf\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.376034 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.375992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:11.476754 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.476715    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:11.630756 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.630728    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"]
Aug 04 18:35:11.633382 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:35:11.633354    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779885fa_3b76_4d94_9957_dca6b1b0d1c5.slice/crio-a105257e99661a749809d3a01d43b5f0ce4961be733ec8184e58d3c1d1ca5597 WatchSource:0}: Error finding container a105257e99661a749809d3a01d43b5f0ce4961be733ec8184e58d3c1d1ca5597: Status 404 returned error can't find the container with id a105257e99661a749809d3a01d43b5f0ce4961be733ec8184e58d3c1d1ca5597
Aug 04 18:35:11.935459 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.935367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv" event={"ID":"931c40f8-d384-454c-858f-9efa49d5b736","Type":"ContainerDied","Data":"77b7c4d46e9290c3ccc592535e2a84e093315a360dde6b318af712c9bbfa88a8"}
Aug 04 18:35:11.935459 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.935410    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"
Aug 04 18:35:11.935459 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.935445    2581 scope.go:117] "RemoveContainer" containerID="2bdb78afd61e923ed3420855aa5203316c3a0bd2a3532291ba28b5260e590cd8"
Aug 04 18:35:11.937491 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.937458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" event={"ID":"779885fa-3b76-4d94-9957-dca6b1b0d1c5","Type":"ContainerStarted","Data":"8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320"}
Aug 04 18:35:11.937616 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.937538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" event={"ID":"779885fa-3b76-4d94-9957-dca6b1b0d1c5","Type":"ContainerStarted","Data":"a105257e99661a749809d3a01d43b5f0ce4961be733ec8184e58d3c1d1ca5597"}
Aug 04 18:35:11.944150 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.944126    2581 scope.go:117] "RemoveContainer" containerID="f80beb104318c092e0c711c73db9deec428d6e7d3408d263f7dbf440de2ed520"
Aug 04 18:35:11.952574 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.952547    2581 scope.go:117] "RemoveContainer" containerID="966c3a70302296b74857937af517f3fee23ede2241727f2bcd1f2017182d1b30"
Aug 04 18:35:11.958312 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.958268    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"]
Aug 04 18:35:11.965952 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:11.965922    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-4b7kv"]
Aug 04 18:35:13.214989 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:13.214950    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="931c40f8-d384-454c-858f-9efa49d5b736" path="/var/lib/kubelet/pods/931c40f8-d384-454c-858f-9efa49d5b736/volumes"
Aug 04 18:35:15.956298 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:15.956262    2581 generic.go:356] "Generic (PLEG): container finished" podID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerID="8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320" exitCode=0
Aug 04 18:35:15.956735 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:15.956322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" event={"ID":"779885fa-3b76-4d94-9957-dca6b1b0d1c5","Type":"ContainerDied","Data":"8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320"}
Aug 04 18:35:16.962800 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:16.962760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" event={"ID":"779885fa-3b76-4d94-9957-dca6b1b0d1c5","Type":"ContainerStarted","Data":"bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0"}
Aug 04 18:35:16.963327 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:16.962835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" event={"ID":"779885fa-3b76-4d94-9957-dca6b1b0d1c5","Type":"ContainerStarted","Data":"94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39"}
Aug 04 18:35:16.963327 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:16.963075    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:16.992103 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:16.988687    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podStartSLOduration=2.373795867 podStartE2EDuration="5.988667396s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.614871529s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:35:16.98667229 +0000 UTC m=+2142.331073448" watchObservedRunningTime="2026-08-04 18:35:16.988667396 +0000 UTC m=+2142.333068554"
Aug 04 18:35:17.967475 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:17.967437    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:17.969000 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:17.968961    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:35:18.971519 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:18.971469    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:35:23.976808 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:23.976768    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:35:23.977512 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:23.977482    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:35:33.978021 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:33.977935    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:35:43.977594 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:43.977549    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:35:52.210105 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:52.210073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:53.977633 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:53.977592    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:35:57.393316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:57.393281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:57.743318 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:35:57.743240    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:03.977479 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:03.977435    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:36:13.977712 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:13.977669    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:36:16.940426 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:16.940393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:23.978060 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:23.978001    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:36:26.290505 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:26.290466    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:33.978520 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:33.978488    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:36:36.261497 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:36.261466    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:38.613445 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:38.613411    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:41.006218 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:41.006188    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:43.748270 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.748231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:43.777032 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.776996    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"]
Aug 04 18:36:43.780122 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.780099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.782527 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.782487    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Aug 04 18:36:43.782693 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.782532    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Aug 04 18:36:43.793434 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.793403    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"]
Aug 04 18:36:43.892640 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.892595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lnjv\" (UniqueName: \"kubernetes.io/projected/276e81e0-215c-4e5b-b41c-589062be76c9-kube-api-access-4lnjv\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.892851 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.892670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/276e81e0-215c-4e5b-b41c-589062be76c9-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.892851 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.892770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.892851 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.892793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/276e81e0-215c-4e5b-b41c-589062be76c9-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.892851 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.892820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.930240 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.930187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:36:43.993486 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.993442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.993486 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.993483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/276e81e0-215c-4e5b-b41c-589062be76c9-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.993791 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.993513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.993791 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.993547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lnjv\" (UniqueName: \"kubernetes.io/projected/276e81e0-215c-4e5b-b41c-589062be76c9-kube-api-access-4lnjv\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.993791 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.993572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/276e81e0-215c-4e5b-b41c-589062be76c9-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.993944 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.993916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/276e81e0-215c-4e5b-b41c-589062be76c9-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.994294 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.994252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.994385 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.994314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:43.996290 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:43.996272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/276e81e0-215c-4e5b-b41c-589062be76c9-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:44.003993 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:44.003925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lnjv\" (UniqueName: \"kubernetes.io/projected/276e81e0-215c-4e5b-b41c-589062be76c9-kube-api-access-4lnjv\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:44.091035 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:44.090994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:44.237444 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:44.237408    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"]
Aug 04 18:36:44.241352 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:36:44.241319    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276e81e0_215c_4e5b_b41c_589062be76c9.slice/crio-f0ba028f81b517607cc7ca4019cd22befa12d09e8ccce80ffdb6e1f8e8ab957e WatchSource:0}: Error finding container f0ba028f81b517607cc7ca4019cd22befa12d09e8ccce80ffdb6e1f8e8ab957e: Status 404 returned error can't find the container with id f0ba028f81b517607cc7ca4019cd22befa12d09e8ccce80ffdb6e1f8e8ab957e
Aug 04 18:36:44.312767 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:44.312724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" event={"ID":"276e81e0-215c-4e5b-b41c-589062be76c9","Type":"ContainerStarted","Data":"88ee311cfc538d13ef0f7f72fb64b0748bf3a3b6160af0379fd40c40a55fddc2"}
Aug 04 18:36:44.312942 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:44.312782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" event={"ID":"276e81e0-215c-4e5b-b41c-589062be76c9","Type":"ContainerStarted","Data":"f0ba028f81b517607cc7ca4019cd22befa12d09e8ccce80ffdb6e1f8e8ab957e"}
Aug 04 18:36:49.333479 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:49.333444    2581 generic.go:356] "Generic (PLEG): container finished" podID="276e81e0-215c-4e5b-b41c-589062be76c9" containerID="88ee311cfc538d13ef0f7f72fb64b0748bf3a3b6160af0379fd40c40a55fddc2" exitCode=0
Aug 04 18:36:49.333860 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:49.333518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" event={"ID":"276e81e0-215c-4e5b-b41c-589062be76c9","Type":"ContainerDied","Data":"88ee311cfc538d13ef0f7f72fb64b0748bf3a3b6160af0379fd40c40a55fddc2"}
Aug 04 18:36:54.358186 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:54.358147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" event={"ID":"276e81e0-215c-4e5b-b41c-589062be76c9","Type":"ContainerStarted","Data":"f8142af95393afc8e6fa73c437c79f7972569b943134dea50d5e245761a6640a"}
Aug 04 18:36:54.358788 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:54.358263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" event={"ID":"276e81e0-215c-4e5b-b41c-589062be76c9","Type":"ContainerStarted","Data":"b6a19c91488b8ef51c3daf05f73273b7bea60453ba13df5673894f576d36c900"}
Aug 04 18:36:54.358788 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:54.358494    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:54.380388 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:54.380334    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podStartSLOduration=2.629938568 podStartE2EDuration="11.380316416s" totalImagesPullingTime="4.04687958s" totalInitContainerRuntime="4.703498268s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:36:54.379402001 +0000 UTC m=+2239.723803158" watchObservedRunningTime="2026-08-04 18:36:54.380316416 +0000 UTC m=+2239.724717572"
Aug 04 18:36:55.362647 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:55.362609    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:36:55.364015 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:55.363977    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:36:56.367028 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:36:56.366933    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:37:01.372031 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:01.371998    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:37:01.372648 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:01.372620    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:37:11.373578 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:11.373545    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:37:21.469557 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:21.469509    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:25.554368 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:25.554333    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"]
Aug 04 18:37:25.554878 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:25.554645    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kserve-container" containerID="cri-o://b6a19c91488b8ef51c3daf05f73273b7bea60453ba13df5673894f576d36c900" gracePeriod=30
Aug 04 18:37:25.554878 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:25.554742    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kube-rbac-proxy" containerID="cri-o://f8142af95393afc8e6fa73c437c79f7972569b943134dea50d5e245761a6640a" gracePeriod=30
Aug 04 18:37:25.581965 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:25.581916    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:26.367640 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:26.367598    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Aug 04 18:37:26.495223 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:26.495179    2581 generic.go:356] "Generic (PLEG): container finished" podID="276e81e0-215c-4e5b-b41c-589062be76c9" containerID="f8142af95393afc8e6fa73c437c79f7972569b943134dea50d5e245761a6640a" exitCode=2
Aug 04 18:37:26.495419 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:26.495261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" event={"ID":"276e81e0-215c-4e5b-b41c-589062be76c9","Type":"ContainerDied","Data":"f8142af95393afc8e6fa73c437c79f7972569b943134dea50d5e245761a6640a"}
Aug 04 18:37:30.695151 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:30.695110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:30.929986 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:30.929945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:37:31.367699 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:31.367656    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Aug 04 18:37:36.368130 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:36.368074    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Aug 04 18:37:36.368539 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:36.368223    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:37:41.367778 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:41.367731    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Aug 04 18:37:46.367480 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:46.367433    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Aug 04 18:37:51.367659 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:51.367612    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Aug 04 18:37:55.622576 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:55.622542    2581 generic.go:356] "Generic (PLEG): container finished" podID="276e81e0-215c-4e5b-b41c-589062be76c9" containerID="b6a19c91488b8ef51c3daf05f73273b7bea60453ba13df5673894f576d36c900" exitCode=137
Aug 04 18:37:55.622965 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:55.622598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" event={"ID":"276e81e0-215c-4e5b-b41c-589062be76c9","Type":"ContainerDied","Data":"b6a19c91488b8ef51c3daf05f73273b7bea60453ba13df5673894f576d36c900"}
Aug 04 18:37:56.204251 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.204224    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:37:56.296185 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.296136    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-cabundle-cert\") pod \"276e81e0-215c-4e5b-b41c-589062be76c9\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") "
Aug 04 18:37:56.296185 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.296179    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-kube-rbac-proxy-sar-config\") pod \"276e81e0-215c-4e5b-b41c-589062be76c9\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") "
Aug 04 18:37:56.296417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.296255    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/276e81e0-215c-4e5b-b41c-589062be76c9-kserve-provision-location\") pod \"276e81e0-215c-4e5b-b41c-589062be76c9\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") "
Aug 04 18:37:56.296417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.296314    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/276e81e0-215c-4e5b-b41c-589062be76c9-proxy-tls\") pod \"276e81e0-215c-4e5b-b41c-589062be76c9\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") "
Aug 04 18:37:56.296417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.296370    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lnjv\" (UniqueName: \"kubernetes.io/projected/276e81e0-215c-4e5b-b41c-589062be76c9-kube-api-access-4lnjv\") pod \"276e81e0-215c-4e5b-b41c-589062be76c9\" (UID: \"276e81e0-215c-4e5b-b41c-589062be76c9\") "
Aug 04 18:37:56.296623 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.296604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "276e81e0-215c-4e5b-b41c-589062be76c9" (UID: "276e81e0-215c-4e5b-b41c-589062be76c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:37:56.296661 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.296635    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "276e81e0-215c-4e5b-b41c-589062be76c9" (UID: "276e81e0-215c-4e5b-b41c-589062be76c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:37:56.298655 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.298618    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276e81e0-215c-4e5b-b41c-589062be76c9-kube-api-access-4lnjv" (OuterVolumeSpecName: "kube-api-access-4lnjv") pod "276e81e0-215c-4e5b-b41c-589062be76c9" (UID: "276e81e0-215c-4e5b-b41c-589062be76c9"). InnerVolumeSpecName "kube-api-access-4lnjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:37:56.298655 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.298641    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276e81e0-215c-4e5b-b41c-589062be76c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "276e81e0-215c-4e5b-b41c-589062be76c9" (UID: "276e81e0-215c-4e5b-b41c-589062be76c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:37:56.306557 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.306524    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276e81e0-215c-4e5b-b41c-589062be76c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "276e81e0-215c-4e5b-b41c-589062be76c9" (UID: "276e81e0-215c-4e5b-b41c-589062be76c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:37:56.396998 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.396903    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/276e81e0-215c-4e5b-b41c-589062be76c9-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:37:56.396998 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.396934    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/276e81e0-215c-4e5b-b41c-589062be76c9-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:37:56.396998 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.396946    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lnjv\" (UniqueName: \"kubernetes.io/projected/276e81e0-215c-4e5b-b41c-589062be76c9-kube-api-access-4lnjv\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:37:56.396998 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.396955    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:37:56.396998 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.396966    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/276e81e0-215c-4e5b-b41c-589062be76c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:37:56.628640 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.628607    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"
Aug 04 18:37:56.628640 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.628624    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw" event={"ID":"276e81e0-215c-4e5b-b41c-589062be76c9","Type":"ContainerDied","Data":"f0ba028f81b517607cc7ca4019cd22befa12d09e8ccce80ffdb6e1f8e8ab957e"}
Aug 04 18:37:56.629214 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.628688    2581 scope.go:117] "RemoveContainer" containerID="f8142af95393afc8e6fa73c437c79f7972569b943134dea50d5e245761a6640a"
Aug 04 18:37:56.638127 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.638102    2581 scope.go:117] "RemoveContainer" containerID="b6a19c91488b8ef51c3daf05f73273b7bea60453ba13df5673894f576d36c900"
Aug 04 18:37:56.646323 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.646301    2581 scope.go:117] "RemoveContainer" containerID="88ee311cfc538d13ef0f7f72fb64b0748bf3a3b6160af0379fd40c40a55fddc2"
Aug 04 18:37:56.659418 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.659383    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"]
Aug 04 18:37:56.662862 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:56.662827    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-2mzfw"]
Aug 04 18:37:57.214710 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:37:57.214673    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" path="/var/lib/kubelet/pods/276e81e0-215c-4e5b-b41c-589062be76c9/volumes"
Aug 04 18:38:21.139398 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:21.139357    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"]
Aug 04 18:38:21.139879 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:21.139723    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container" containerID="cri-o://94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39" gracePeriod=30
Aug 04 18:38:21.139879 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:21.139736    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kube-rbac-proxy" containerID="cri-o://bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0" gracePeriod=30
Aug 04 18:38:21.153747 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:21.153718    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:21.737555 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:21.737521    2581 generic.go:356] "Generic (PLEG): container finished" podID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerID="bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0" exitCode=2
Aug 04 18:38:21.737728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:21.737590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" event={"ID":"779885fa-3b76-4d94-9957-dca6b1b0d1c5","Type":"ContainerDied","Data":"bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0"}
Aug 04 18:38:23.971897 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:23.971850    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Aug 04 18:38:23.977687 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:23.977650    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:38:26.197441 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.197412    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:38:26.237972 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.237880    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-rbac-proxy-sar-config\") pod \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") "
Aug 04 18:38:26.237972 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.237970    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kserve-provision-location\") pod \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") "
Aug 04 18:38:26.238234 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.237999    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779885fa-3b76-4d94-9957-dca6b1b0d1c5-proxy-tls\") pod \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") "
Aug 04 18:38:26.238234 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.238070    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp7sc\" (UniqueName: \"kubernetes.io/projected/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-api-access-qp7sc\") pod \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") "
Aug 04 18:38:26.238234 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.238102    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-cabundle-cert\") pod \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\" (UID: \"779885fa-3b76-4d94-9957-dca6b1b0d1c5\") "
Aug 04 18:38:26.238414 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.238290    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "779885fa-3b76-4d94-9957-dca6b1b0d1c5" (UID: "779885fa-3b76-4d94-9957-dca6b1b0d1c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:38:26.238414 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.238381    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "779885fa-3b76-4d94-9957-dca6b1b0d1c5" (UID: "779885fa-3b76-4d94-9957-dca6b1b0d1c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:38:26.239540 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.238551    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "779885fa-3b76-4d94-9957-dca6b1b0d1c5" (UID: "779885fa-3b76-4d94-9957-dca6b1b0d1c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:38:26.239540 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.239013    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:38:26.239540 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.239034    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:38:26.239540 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.239071    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:38:26.240921 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.240889    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-api-access-qp7sc" (OuterVolumeSpecName: "kube-api-access-qp7sc") pod "779885fa-3b76-4d94-9957-dca6b1b0d1c5" (UID: "779885fa-3b76-4d94-9957-dca6b1b0d1c5"). InnerVolumeSpecName "kube-api-access-qp7sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:38:26.241824 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.241802    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779885fa-3b76-4d94-9957-dca6b1b0d1c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "779885fa-3b76-4d94-9957-dca6b1b0d1c5" (UID: "779885fa-3b76-4d94-9957-dca6b1b0d1c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:38:26.295058 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.295019    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:26.326591 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326550    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"]
Aug 04 18:38:26.326847 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326835    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="storage-initializer"
Aug 04 18:38:26.326902 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326849    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="storage-initializer"
Aug 04 18:38:26.326902 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326861    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="storage-initializer"
Aug 04 18:38:26.326902 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326867    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="storage-initializer"
Aug 04 18:38:26.326902 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326881    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container"
Aug 04 18:38:26.326902 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326886    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container"
Aug 04 18:38:26.326902 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326892    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kube-rbac-proxy"
Aug 04 18:38:26.326902 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326897    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kube-rbac-proxy"
Aug 04 18:38:26.326902 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326904    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kube-rbac-proxy"
Aug 04 18:38:26.327157 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326911    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kube-rbac-proxy"
Aug 04 18:38:26.327157 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326916    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kserve-container"
Aug 04 18:38:26.327157 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326921    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kserve-container"
Aug 04 18:38:26.327157 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326969    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kube-rbac-proxy"
Aug 04 18:38:26.327157 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326976    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kserve-container"
Aug 04 18:38:26.327157 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326982    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerName="kserve-container"
Aug 04 18:38:26.327157 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.326989    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="276e81e0-215c-4e5b-b41c-589062be76c9" containerName="kube-rbac-proxy"
Aug 04 18:38:26.330137 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.330114    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.332145 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.332111    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Aug 04 18:38:26.332286 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.332231    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Aug 04 18:38:26.340147 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.340125    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779885fa-3b76-4d94-9957-dca6b1b0d1c5-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:38:26.340260 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.340172    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7sc\" (UniqueName: \"kubernetes.io/projected/779885fa-3b76-4d94-9957-dca6b1b0d1c5-kube-api-access-qp7sc\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:38:26.347465 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.347436    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"]
Aug 04 18:38:26.380092 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.380055    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:26.441385 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.441349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.441597 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.441404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.441597 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.441505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v85r8\" (UniqueName: \"kubernetes.io/projected/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-api-access-v85r8\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.441597 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.441556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-cabundle-cert\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.441735 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.441626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-proxy-tls\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.542640 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.542596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-proxy-tls\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.542930 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.542652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.542930 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.542685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.542930 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.542733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v85r8\" (UniqueName: \"kubernetes.io/projected/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-api-access-v85r8\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.542930 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:38:26.542766    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Aug 04 18:38:26.542930 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:38:26.542850    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-proxy-tls podName:7066cda2-29f2-4da6-8daa-1ccc29f5e3e5 nodeName:}" failed. No retries permitted until 2026-08-04 18:38:27.042827623 +0000 UTC m=+2332.387228758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-proxy-tls") pod "isvc-canary-stop-predictor-57599f75db-nqljd" (UID: "7066cda2-29f2-4da6-8daa-1ccc29f5e3e5") : secret "isvc-canary-stop-predictor-serving-cert" not found
Aug 04 18:38:26.542930 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.542773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-cabundle-cert\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.543274 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.543217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.543463 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.543444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.543499 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.543470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-cabundle-cert\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.552968 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.552944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v85r8\" (UniqueName: \"kubernetes.io/projected/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-api-access-v85r8\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:26.681402 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.681336    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:26.759218 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.759186    2581 generic.go:356] "Generic (PLEG): container finished" podID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" containerID="94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39" exitCode=0
Aug 04 18:38:26.759403 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.759241    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"
Aug 04 18:38:26.759403 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.759299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" event={"ID":"779885fa-3b76-4d94-9957-dca6b1b0d1c5","Type":"ContainerDied","Data":"94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39"}
Aug 04 18:38:26.759403 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.759349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf" event={"ID":"779885fa-3b76-4d94-9957-dca6b1b0d1c5","Type":"ContainerDied","Data":"a105257e99661a749809d3a01d43b5f0ce4961be733ec8184e58d3c1d1ca5597"}
Aug 04 18:38:26.759403 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.759379    2581 scope.go:117] "RemoveContainer" containerID="bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0"
Aug 04 18:38:26.768399 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.768381    2581 scope.go:117] "RemoveContainer" containerID="94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39"
Aug 04 18:38:26.776840 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.776785    2581 scope.go:117] "RemoveContainer" containerID="8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320"
Aug 04 18:38:26.780841 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.780813    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:38:26.786354 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.786315    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"]
Aug 04 18:38:26.789940 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.789879    2581 scope.go:117] "RemoveContainer" containerID="bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0"
Aug 04 18:38:26.790581 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:38:26.790424    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0\": container with ID starting with bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0 not found: ID does not exist" containerID="bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0"
Aug 04 18:38:26.790581 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.790466    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0"} err="failed to get container status \"bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0\": rpc error: code = NotFound desc = could not find container \"bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0\": container with ID starting with bf309ce2fc9a196d5450cbbe4d565334d133ce65b846df95670956e6ccaf17b0 not found: ID does not exist"
Aug 04 18:38:26.790581 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.790494    2581 scope.go:117] "RemoveContainer" containerID="94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39"
Aug 04 18:38:26.790829 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:38:26.790809    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39\": container with ID starting with 94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39 not found: ID does not exist" containerID="94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39"
Aug 04 18:38:26.790868 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.790838    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39"} err="failed to get container status \"94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39\": rpc error: code = NotFound desc = could not find container \"94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39\": container with ID starting with 94b306c29a87d724922e57206dd6d76056231fbbe1559c8e22d07dcaab6ebb39 not found: ID does not exist"
Aug 04 18:38:26.790868 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.790854    2581 scope.go:117] "RemoveContainer" containerID="8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320"
Aug 04 18:38:26.791181 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:38:26.791159    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320\": container with ID starting with 8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320 not found: ID does not exist" containerID="8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320"
Aug 04 18:38:26.791272 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.791189    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320"} err="failed to get container status \"8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320\": rpc error: code = NotFound desc = could not find container \"8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320\": container with ID starting with 8c780b42f2c254984a75d8b497eefdd19f25221d65b12c82e9a23c3507a6d320 not found: ID does not exist"
Aug 04 18:38:26.792357 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:26.792338    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-b82lf"]
Aug 04 18:38:27.047543 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:27.047502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-proxy-tls\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:27.050366 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:27.050327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-proxy-tls\") pod \"isvc-canary-stop-predictor-57599f75db-nqljd\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:27.217080 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:27.216213    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="779885fa-3b76-4d94-9957-dca6b1b0d1c5" path="/var/lib/kubelet/pods/779885fa-3b76-4d94-9957-dca6b1b0d1c5/volumes"
Aug 04 18:38:27.241246 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:27.241199    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:27.383496 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:27.383461    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"]
Aug 04 18:38:27.386769 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:38:27.386737    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7066cda2_29f2_4da6_8daa_1ccc29f5e3e5.slice/crio-182bb2bb2e184ef717370b94665aa6756a043952255ef5bde35dc58877fcde71 WatchSource:0}: Error finding container 182bb2bb2e184ef717370b94665aa6756a043952255ef5bde35dc58877fcde71: Status 404 returned error can't find the container with id 182bb2bb2e184ef717370b94665aa6756a043952255ef5bde35dc58877fcde71
Aug 04 18:38:27.765979 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:27.765874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" event={"ID":"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5","Type":"ContainerStarted","Data":"b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473"}
Aug 04 18:38:27.765979 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:27.765950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" event={"ID":"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5","Type":"ContainerStarted","Data":"182bb2bb2e184ef717370b94665aa6756a043952255ef5bde35dc58877fcde71"}
Aug 04 18:38:31.783726 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:31.783691    2581 generic.go:356] "Generic (PLEG): container finished" podID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerID="b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473" exitCode=0
Aug 04 18:38:31.783726 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:31.783730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" event={"ID":"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5","Type":"ContainerDied","Data":"b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473"}
Aug 04 18:38:32.790277 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:32.790236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" event={"ID":"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5","Type":"ContainerStarted","Data":"5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9"}
Aug 04 18:38:32.790712 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:32.790347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" event={"ID":"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5","Type":"ContainerStarted","Data":"96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6"}
Aug 04 18:38:32.790712 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:32.790559    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:32.812122 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:32.812035    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podStartSLOduration=3.175945392 podStartE2EDuration="6.812019223s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.636073831s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:38:32.810766276 +0000 UTC m=+2338.155167435" watchObservedRunningTime="2026-08-04 18:38:32.812019223 +0000 UTC m=+2338.156420379"
Aug 04 18:38:33.794192 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:33.794161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:33.795584 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:33.795554    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:38:34.798533 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:34.798491    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:38:39.803863 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:39.803833    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:38:39.804394 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:39.804364    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:38:49.804610 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:49.804553    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:38:59.804820 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:38:59.804780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:39:09.804846 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:09.804802    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:39:19.805152 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:19.805106    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:39:29.804886 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:29.804840    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:39:35.296412 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:35.296379    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:39:35.296958 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:35.296814    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:39:35.304180 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:35.304154    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:39:35.304367 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:35.304163    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:39:39.805283 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:39.805232    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:39:43.013326 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:43.013285    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:48.430923 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:48.430877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:48.506490 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:48.506450    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:39:49.805235 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:39:49.805207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:40:51.409232 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:51.409179    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"]
Aug 04 18:40:51.409833 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:51.409523    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container" containerID="cri-o://96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6" gracePeriod=30
Aug 04 18:40:51.409833 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:51.409680    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kube-rbac-proxy" containerID="cri-o://5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9" gracePeriod=30
Aug 04 18:40:51.424672 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:51.424611    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:51.524219 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:51.524182    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:52.357834 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:52.357789    2581 generic.go:356] "Generic (PLEG): container finished" podID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerID="5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9" exitCode=2
Aug 04 18:40:52.358024 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:52.357826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" event={"ID":"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5","Type":"ContainerDied","Data":"5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9"}
Aug 04 18:40:54.798717 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:54.798676    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Aug 04 18:40:56.356786 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.356758    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:40:56.376059 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.376003    2581 generic.go:356] "Generic (PLEG): container finished" podID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerID="96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6" exitCode=0
Aug 04 18:40:56.376215 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.376073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" event={"ID":"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5","Type":"ContainerDied","Data":"96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6"}
Aug 04 18:40:56.376215 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.376088    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"
Aug 04 18:40:56.376215 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.376143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd" event={"ID":"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5","Type":"ContainerDied","Data":"182bb2bb2e184ef717370b94665aa6756a043952255ef5bde35dc58877fcde71"}
Aug 04 18:40:56.376215 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.376170    2581 scope.go:117] "RemoveContainer" containerID="5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9"
Aug 04 18:40:56.385751 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.385720    2581 scope.go:117] "RemoveContainer" containerID="96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6"
Aug 04 18:40:56.394678 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.394652    2581 scope.go:117] "RemoveContainer" containerID="b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473"
Aug 04 18:40:56.404578 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.404553    2581 scope.go:117] "RemoveContainer" containerID="5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9"
Aug 04 18:40:56.404944 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:40:56.404921    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9\": container with ID starting with 5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9 not found: ID does not exist" containerID="5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9"
Aug 04 18:40:56.405091 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.404953    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9"} err="failed to get container status \"5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9\": rpc error: code = NotFound desc = could not find container \"5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9\": container with ID starting with 5f337cddaae9ca3d8b25fa2df6c648cce018d2164275bff20645b1e138ad37e9 not found: ID does not exist"
Aug 04 18:40:56.405091 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.404983    2581 scope.go:117] "RemoveContainer" containerID="96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6"
Aug 04 18:40:56.405403 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:40:56.405378    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6\": container with ID starting with 96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6 not found: ID does not exist" containerID="96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6"
Aug 04 18:40:56.405487 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.405412    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6"} err="failed to get container status \"96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6\": rpc error: code = NotFound desc = could not find container \"96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6\": container with ID starting with 96b34764e75410a62bcc671a80c3ab3725cd1b6e2334b2fd041626c14af4edd6 not found: ID does not exist"
Aug 04 18:40:56.405487 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.405431    2581 scope.go:117] "RemoveContainer" containerID="b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473"
Aug 04 18:40:56.405800 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:40:56.405778    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473\": container with ID starting with b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473 not found: ID does not exist" containerID="b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473"
Aug 04 18:40:56.405952 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.405805    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473"} err="failed to get container status \"b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473\": rpc error: code = NotFound desc = could not find container \"b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473\": container with ID starting with b7f37c941d6cad179fcf959e0dea2718cc8e76ad9e2649354bd93563135f1473 not found: ID does not exist"
Aug 04 18:40:56.453300 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.453203    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v85r8\" (UniqueName: \"kubernetes.io/projected/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-api-access-v85r8\") pod \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") "
Aug 04 18:40:56.453300 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.453244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-proxy-tls\") pod \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") "
Aug 04 18:40:56.453300 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.453286    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kserve-provision-location\") pod \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") "
Aug 04 18:40:56.453555 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.453342    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-cabundle-cert\") pod \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") "
Aug 04 18:40:56.453555 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.453359    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-rbac-proxy-sar-config\") pod \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\" (UID: \"7066cda2-29f2-4da6-8daa-1ccc29f5e3e5\") "
Aug 04 18:40:56.453723 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.453694    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" (UID: "7066cda2-29f2-4da6-8daa-1ccc29f5e3e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:40:56.453801 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.453726    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" (UID: "7066cda2-29f2-4da6-8daa-1ccc29f5e3e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:40:56.453860 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.453835    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" (UID: "7066cda2-29f2-4da6-8daa-1ccc29f5e3e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:40:56.455597 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.455566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" (UID: "7066cda2-29f2-4da6-8daa-1ccc29f5e3e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:40:56.455689 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.455632    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-api-access-v85r8" (OuterVolumeSpecName: "kube-api-access-v85r8") pod "7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" (UID: "7066cda2-29f2-4da6-8daa-1ccc29f5e3e5"). InnerVolumeSpecName "kube-api-access-v85r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:40:56.554404 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.554360    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:40:56.554404 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.554397    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:40:56.554404 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.554411    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:40:56.554719 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.554423    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:40:56.554719 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.554435    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v85r8\" (UniqueName: \"kubernetes.io/projected/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5-kube-api-access-v85r8\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:40:56.701482 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.701444    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"]
Aug 04 18:40:56.708470 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:56.708389    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-nqljd"]
Aug 04 18:40:57.216906 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:40:57.216871    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" path="/var/lib/kubelet/pods/7066cda2-29f2-4da6-8daa-1ccc29f5e3e5/volumes"
Aug 04 18:41:01.742350 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.742267    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:01.813469 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.813429    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"]
Aug 04 18:41:01.813764 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.813751    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="storage-initializer"
Aug 04 18:41:01.813808 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.813766    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="storage-initializer"
Aug 04 18:41:01.813808 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.813783    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container"
Aug 04 18:41:01.813808 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.813789    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container"
Aug 04 18:41:01.813808 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.813804    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kube-rbac-proxy"
Aug 04 18:41:01.813932 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.813810    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kube-rbac-proxy"
Aug 04 18:41:01.813932 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.813851    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kube-rbac-proxy"
Aug 04 18:41:01.813932 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.813862    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7066cda2-29f2-4da6-8daa-1ccc29f5e3e5" containerName="kserve-container"
Aug 04 18:41:01.818821 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.818795    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:01.824526 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.824496    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:41:01.824704 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.824559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:41:01.825073 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.825026    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Aug 04 18:41:01.825342 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.825310    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Aug 04 18:41:01.825796 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.825355    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:41:01.825796 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.825712    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:41:01.826506 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.826485    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:41:01.839864 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.839826    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"]
Aug 04 18:41:01.898205 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.898158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34947526-e631-4714-8752-77a7d5694a70-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:01.898389 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.898218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpcn9\" (UniqueName: \"kubernetes.io/projected/34947526-e631-4714-8752-77a7d5694a70-kube-api-access-wpcn9\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:01.898389 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.898243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:01.898389 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.898328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34947526-e631-4714-8752-77a7d5694a70-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:01.898389 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.898359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:01.999488 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.999390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34947526-e631-4714-8752-77a7d5694a70-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:01.999488 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.999430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:01.999488 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.999451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34947526-e631-4714-8752-77a7d5694a70-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:01.999488 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.999489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpcn9\" (UniqueName: \"kubernetes.io/projected/34947526-e631-4714-8752-77a7d5694a70-kube-api-access-wpcn9\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:01.999835 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.999604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:01.999972 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:01.999948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34947526-e631-4714-8752-77a7d5694a70-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:02.000172 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:02.000153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:02.000265 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:02.000159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:02.002220 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:02.002196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34947526-e631-4714-8752-77a7d5694a70-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:02.014343 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:02.014305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpcn9\" (UniqueName: \"kubernetes.io/projected/34947526-e631-4714-8752-77a7d5694a70-kube-api-access-wpcn9\") pod \"isvc-lightgbm-predictor-54f85ddf88-st4pc\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:02.020446 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:02.020416    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:02.129015 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:02.128972    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:02.274385 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:02.274347    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"]
Aug 04 18:41:02.277467 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:41:02.277429    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34947526_e631_4714_8752_77a7d5694a70.slice/crio-c031443d8bb57d24b30a3e2137be34aa66faa61efc2a6142e5af8967750365fe WatchSource:0}: Error finding container c031443d8bb57d24b30a3e2137be34aa66faa61efc2a6142e5af8967750365fe: Status 404 returned error can't find the container with id c031443d8bb57d24b30a3e2137be34aa66faa61efc2a6142e5af8967750365fe
Aug 04 18:41:02.400984 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:02.400948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" event={"ID":"34947526-e631-4714-8752-77a7d5694a70","Type":"ContainerStarted","Data":"cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488"}
Aug 04 18:41:02.401170 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:02.401008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" event={"ID":"34947526-e631-4714-8752-77a7d5694a70","Type":"ContainerStarted","Data":"c031443d8bb57d24b30a3e2137be34aa66faa61efc2a6142e5af8967750365fe"}
Aug 04 18:41:06.417823 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:06.417723    2581 generic.go:356] "Generic (PLEG): container finished" podID="34947526-e631-4714-8752-77a7d5694a70" containerID="cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488" exitCode=0
Aug 04 18:41:06.417823 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:06.417790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" event={"ID":"34947526-e631-4714-8752-77a7d5694a70","Type":"ContainerDied","Data":"cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488"}
Aug 04 18:41:06.419161 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:06.419144    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:41:13.452167 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:13.452116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" event={"ID":"34947526-e631-4714-8752-77a7d5694a70","Type":"ContainerStarted","Data":"c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc"}
Aug 04 18:41:13.452591 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:13.452197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" event={"ID":"34947526-e631-4714-8752-77a7d5694a70","Type":"ContainerStarted","Data":"dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31"}
Aug 04 18:41:13.452591 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:13.452395    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:13.481203 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:13.481143    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podStartSLOduration=2.598323256 podStartE2EDuration="12.481122041s" totalImagesPullingTime="6.089431767s" totalInitContainerRuntime="3.793367018s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:13.479268165 +0000 UTC m=+2498.823669321" watchObservedRunningTime="2026-08-04 18:41:13.481122041 +0000 UTC m=+2498.825523198"
Aug 04 18:41:14.457117 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:14.457082    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:14.458680 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:14.458640    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:41:15.461843 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:15.461790    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:41:20.466409 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:20.466376    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:41:20.466854 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:20.466826    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:41:28.420887 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:28.420786    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:30.467658 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:30.467614    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:41:33.463444 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:33.463407    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:33.704730 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:33.704694    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:41:40.467417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:40.467375    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:41:50.467538 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:41:50.467493    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:42:00.467176 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:00.467134    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:42:10.467608 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:10.467566    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:42:20.467702 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:20.467655    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:42:20.853322 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:20.853284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:24.210203 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:24.210154    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:42:24.381265 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:24.381231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:28.699705 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:28.699651    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:28.776312 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:28.776276    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:34.211297 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:34.211260    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:42:41.871466 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:41.871423    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"]
Aug 04 18:42:41.871869 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:41.871678    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" containerID="cri-o://dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31" gracePeriod=30
Aug 04 18:42:41.871869 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:41.871740    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kube-rbac-proxy" containerID="cri-o://c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc" gracePeriod=30
Aug 04 18:42:41.906024 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:41.905968    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:42.814173 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:42.814139    2581 generic.go:356] "Generic (PLEG): container finished" podID="34947526-e631-4714-8752-77a7d5694a70" containerID="c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc" exitCode=2
Aug 04 18:42:42.814341 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:42.814220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" event={"ID":"34947526-e631-4714-8752-77a7d5694a70","Type":"ContainerDied","Data":"c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc"}
Aug 04 18:42:44.210801 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:44.210755    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:42:45.462192 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:45.462144    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Aug 04 18:42:47.029865 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.029827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:47.090589 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.090552    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"]
Aug 04 18:42:47.094358 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.094322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.097851 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.097801    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Aug 04 18:42:47.099241 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.099216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Aug 04 18:42:47.139072 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.139024    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"]
Aug 04 18:42:47.156118 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.156083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:42:47.187085 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.187021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fed8a32a-215b-44fc-864b-86008b814ab6-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.187297 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.187201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.187297 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.187259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr6hr\" (UniqueName: \"kubernetes.io/projected/fed8a32a-215b-44fc-864b-86008b814ab6-kube-api-access-tr6hr\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.187422 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.187294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fed8a32a-215b-44fc-864b-86008b814ab6-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.187422 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.187325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.287993 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.287906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fed8a32a-215b-44fc-864b-86008b814ab6-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.287993 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.287964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.288247 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.288164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr6hr\" (UniqueName: \"kubernetes.io/projected/fed8a32a-215b-44fc-864b-86008b814ab6-kube-api-access-tr6hr\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.288247 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.288232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fed8a32a-215b-44fc-864b-86008b814ab6-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.288350 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.288265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.288631 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.288604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fed8a32a-215b-44fc-864b-86008b814ab6-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.288734 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.288711    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.288879 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.288859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.290781 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.290758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fed8a32a-215b-44fc-864b-86008b814ab6-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.299854 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.299826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr6hr\" (UniqueName: \"kubernetes.io/projected/fed8a32a-215b-44fc-864b-86008b814ab6-kube-api-access-tr6hr\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.405276 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.405240    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:47.531500 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.531471    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:42:47.562324 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.562229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"]
Aug 04 18:42:47.565830 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:42:47.565806    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed8a32a_215b_44fc_864b_86008b814ab6.slice/crio-9a27f62fb87bd433479c229b282a82c2e4ef8ea2a0f329df5340fce1c55271dd WatchSource:0}: Error finding container 9a27f62fb87bd433479c229b282a82c2e4ef8ea2a0f329df5340fce1c55271dd: Status 404 returned error can't find the container with id 9a27f62fb87bd433479c229b282a82c2e4ef8ea2a0f329df5340fce1c55271dd
Aug 04 18:42:47.591206 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.591174    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpcn9\" (UniqueName: \"kubernetes.io/projected/34947526-e631-4714-8752-77a7d5694a70-kube-api-access-wpcn9\") pod \"34947526-e631-4714-8752-77a7d5694a70\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") "
Aug 04 18:42:47.591371 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.591224    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-cabundle-cert\") pod \"34947526-e631-4714-8752-77a7d5694a70\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") "
Aug 04 18:42:47.591426 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.591402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34947526-e631-4714-8752-77a7d5694a70-kserve-provision-location\") pod \"34947526-e631-4714-8752-77a7d5694a70\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") "
Aug 04 18:42:47.591468 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.591437    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-kube-rbac-proxy-sar-config\") pod \"34947526-e631-4714-8752-77a7d5694a70\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") "
Aug 04 18:42:47.591468 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.591462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34947526-e631-4714-8752-77a7d5694a70-proxy-tls\") pod \"34947526-e631-4714-8752-77a7d5694a70\" (UID: \"34947526-e631-4714-8752-77a7d5694a70\") "
Aug 04 18:42:47.591706 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.591587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34947526-e631-4714-8752-77a7d5694a70" (UID: "34947526-e631-4714-8752-77a7d5694a70"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:42:47.591793 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.591725    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34947526-e631-4714-8752-77a7d5694a70" (UID: "34947526-e631-4714-8752-77a7d5694a70"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:42:47.591793 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.591728    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34947526-e631-4714-8752-77a7d5694a70-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34947526-e631-4714-8752-77a7d5694a70" (UID: "34947526-e631-4714-8752-77a7d5694a70"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:42:47.593565 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.593542    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34947526-e631-4714-8752-77a7d5694a70-kube-api-access-wpcn9" (OuterVolumeSpecName: "kube-api-access-wpcn9") pod "34947526-e631-4714-8752-77a7d5694a70" (UID: "34947526-e631-4714-8752-77a7d5694a70"). InnerVolumeSpecName "kube-api-access-wpcn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:42:47.593655 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.593603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34947526-e631-4714-8752-77a7d5694a70-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34947526-e631-4714-8752-77a7d5694a70" (UID: "34947526-e631-4714-8752-77a7d5694a70"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:42:47.692348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.692304    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpcn9\" (UniqueName: \"kubernetes.io/projected/34947526-e631-4714-8752-77a7d5694a70-kube-api-access-wpcn9\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:42:47.692348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.692334    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:42:47.692348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.692343    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34947526-e631-4714-8752-77a7d5694a70-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:42:47.692348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.692353    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34947526-e631-4714-8752-77a7d5694a70-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:42:47.692348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.692361    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34947526-e631-4714-8752-77a7d5694a70-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:42:47.837698 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.837602    2581 generic.go:356] "Generic (PLEG): container finished" podID="34947526-e631-4714-8752-77a7d5694a70" containerID="dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31" exitCode=0
Aug 04 18:42:47.837698 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.837676    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"
Aug 04 18:42:47.837939 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.837690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" event={"ID":"34947526-e631-4714-8752-77a7d5694a70","Type":"ContainerDied","Data":"dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31"}
Aug 04 18:42:47.837939 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.837805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc" event={"ID":"34947526-e631-4714-8752-77a7d5694a70","Type":"ContainerDied","Data":"c031443d8bb57d24b30a3e2137be34aa66faa61efc2a6142e5af8967750365fe"}
Aug 04 18:42:47.837939 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.837847    2581 scope.go:117] "RemoveContainer" containerID="c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc"
Aug 04 18:42:47.839492 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.839462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" event={"ID":"fed8a32a-215b-44fc-864b-86008b814ab6","Type":"ContainerStarted","Data":"c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905"}
Aug 04 18:42:47.839617 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.839529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" event={"ID":"fed8a32a-215b-44fc-864b-86008b814ab6","Type":"ContainerStarted","Data":"9a27f62fb87bd433479c229b282a82c2e4ef8ea2a0f329df5340fce1c55271dd"}
Aug 04 18:42:47.847319 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.847298    2581 scope.go:117] "RemoveContainer" containerID="dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31"
Aug 04 18:42:47.857302 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.857282    2581 scope.go:117] "RemoveContainer" containerID="cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488"
Aug 04 18:42:47.865453 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.865430    2581 scope.go:117] "RemoveContainer" containerID="c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc"
Aug 04 18:42:47.865827 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:42:47.865806    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc\": container with ID starting with c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc not found: ID does not exist" containerID="c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc"
Aug 04 18:42:47.865891 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.865839    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc"} err="failed to get container status \"c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc\": rpc error: code = NotFound desc = could not find container \"c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc\": container with ID starting with c47504ffdea181db6f63576ed6660f6358fca9c566e894d52a5039b4ea2d32fc not found: ID does not exist"
Aug 04 18:42:47.865891 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.865859    2581 scope.go:117] "RemoveContainer" containerID="dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31"
Aug 04 18:42:47.866160 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:42:47.866144    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31\": container with ID starting with dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31 not found: ID does not exist" containerID="dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31"
Aug 04 18:42:47.866207 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.866164    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31"} err="failed to get container status \"dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31\": rpc error: code = NotFound desc = could not find container \"dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31\": container with ID starting with dea203b86ecfd53abcb72b4e4d4cc2b2f763163e9a2d19af98fb52db8b522d31 not found: ID does not exist"
Aug 04 18:42:47.866207 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.866179    2581 scope.go:117] "RemoveContainer" containerID="cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488"
Aug 04 18:42:47.866437 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:42:47.866425    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488\": container with ID starting with cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488 not found: ID does not exist" containerID="cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488"
Aug 04 18:42:47.866474 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.866440    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488"} err="failed to get container status \"cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488\": rpc error: code = NotFound desc = could not find container \"cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488\": container with ID starting with cf28bd9d3d6caebfd39ffbe7bb020f7388e1da6d3cd597716a1a03c50aa8a488 not found: ID does not exist"
Aug 04 18:42:47.885587 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.885546    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"]
Aug 04 18:42:47.890770 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:47.890736    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-st4pc"]
Aug 04 18:42:49.214525 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:49.214492    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34947526-e631-4714-8752-77a7d5694a70" path="/var/lib/kubelet/pods/34947526-e631-4714-8752-77a7d5694a70/volumes"
Aug 04 18:42:51.859881 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:51.859828    2581 generic.go:356] "Generic (PLEG): container finished" podID="fed8a32a-215b-44fc-864b-86008b814ab6" containerID="c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905" exitCode=0
Aug 04 18:42:51.860393 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:51.860238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" event={"ID":"fed8a32a-215b-44fc-864b-86008b814ab6","Type":"ContainerDied","Data":"c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905"}
Aug 04 18:42:52.866903 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:52.866866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" event={"ID":"fed8a32a-215b-44fc-864b-86008b814ab6","Type":"ContainerStarted","Data":"c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030"}
Aug 04 18:42:52.867432 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:52.866942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" event={"ID":"fed8a32a-215b-44fc-864b-86008b814ab6","Type":"ContainerStarted","Data":"99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24"}
Aug 04 18:42:52.867432 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:52.867193    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:52.867432 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:52.867248    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:52.868735 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:52.868704    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:42:52.892222 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:52.892163    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podStartSLOduration=2.3449300490000002 podStartE2EDuration="5.892142509s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.54721246s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:42:52.890350245 +0000 UTC m=+2598.234751399" watchObservedRunningTime="2026-08-04 18:42:52.892142509 +0000 UTC m=+2598.236543766"
Aug 04 18:42:53.871745 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:53.871701    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:42:58.876613 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:58.876537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:42:58.877170 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:42:58.877141    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:43:08.877475 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:08.877418    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:43:18.841986 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:18.841947    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:18.877786 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:18.877743    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:43:23.938294 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.938244    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:23.979730 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.979698    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"]
Aug 04 18:43:23.980020 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.980006    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kube-rbac-proxy"
Aug 04 18:43:23.980105 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.980021    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kube-rbac-proxy"
Aug 04 18:43:23.980105 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.980033    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container"
Aug 04 18:43:23.980105 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.980056    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container"
Aug 04 18:43:23.980105 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.980064    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="storage-initializer"
Aug 04 18:43:23.980105 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.980069    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="storage-initializer"
Aug 04 18:43:23.980283 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.980112    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kserve-container"
Aug 04 18:43:23.980283 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.980122    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34947526-e631-4714-8752-77a7d5694a70" containerName="kube-rbac-proxy"
Aug 04 18:43:23.983500 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.983468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:23.985859 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.985822    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Aug 04 18:43:23.986006 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.985876    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Aug 04 18:43:23.994236 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:23.994202    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"]
Aug 04 18:43:24.020248 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.020208    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:24.092154 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.092124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.092357 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.092161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.092357 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.092184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc3819e-397f-490e-8125-adb8b62445fc-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.092357 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.092297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc3819e-397f-490e-8125-adb8b62445fc-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.092606 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.092396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcz49\" (UniqueName: \"kubernetes.io/projected/cbc3819e-397f-490e-8125-adb8b62445fc-kube-api-access-qcz49\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.193713 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.193613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcz49\" (UniqueName: \"kubernetes.io/projected/cbc3819e-397f-490e-8125-adb8b62445fc-kube-api-access-qcz49\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.193713 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.193659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.193713 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.193682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.193713 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.193702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc3819e-397f-490e-8125-adb8b62445fc-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.194086 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.193743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc3819e-397f-490e-8125-adb8b62445fc-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.194248 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.194216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc3819e-397f-490e-8125-adb8b62445fc-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.194393 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.194374    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.194606 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.194588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.196514 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.196493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc3819e-397f-490e-8125-adb8b62445fc-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.204020 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.203993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcz49\" (UniqueName: \"kubernetes.io/projected/cbc3819e-397f-490e-8125-adb8b62445fc-kube-api-access-qcz49\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-c5hhf\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.296708 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.296666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:24.441128 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.441099    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"]
Aug 04 18:43:24.444000 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:43:24.443932    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc3819e_397f_490e_8125_adb8b62445fc.slice/crio-ad3269218c74a41748a9cc44a5fd883ab1082fd34607255a978e81b38654cde5 WatchSource:0}: Error finding container ad3269218c74a41748a9cc44a5fd883ab1082fd34607255a978e81b38654cde5: Status 404 returned error can't find the container with id ad3269218c74a41748a9cc44a5fd883ab1082fd34607255a978e81b38654cde5
Aug 04 18:43:24.999821 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.999778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" event={"ID":"cbc3819e-397f-490e-8125-adb8b62445fc","Type":"ContainerStarted","Data":"dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181"}
Aug 04 18:43:25.000244 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:24.999846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" event={"ID":"cbc3819e-397f-490e-8125-adb8b62445fc","Type":"ContainerStarted","Data":"ad3269218c74a41748a9cc44a5fd883ab1082fd34607255a978e81b38654cde5"}
Aug 04 18:43:28.877991 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:28.877945    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:43:29.019364 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:29.019320    2581 generic.go:356] "Generic (PLEG): container finished" podID="cbc3819e-397f-490e-8125-adb8b62445fc" containerID="dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181" exitCode=0
Aug 04 18:43:29.019364 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:29.019367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" event={"ID":"cbc3819e-397f-490e-8125-adb8b62445fc","Type":"ContainerDied","Data":"dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181"}
Aug 04 18:43:38.877015 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:38.876971    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:43:48.877315 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:48.877271    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:43:50.119582 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:50.119542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" event={"ID":"cbc3819e-397f-490e-8125-adb8b62445fc","Type":"ContainerStarted","Data":"ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534"}
Aug 04 18:43:50.119978 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:50.119603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" event={"ID":"cbc3819e-397f-490e-8125-adb8b62445fc","Type":"ContainerStarted","Data":"9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e"}
Aug 04 18:43:50.119978 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:50.119944    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:50.120110 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:50.120095    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:50.121487 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:50.121457    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:43:50.154872 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:50.154809    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podStartSLOduration=3.307165566 podStartE2EDuration="27.154793124s" totalImagesPullingTime="20.058460663s" totalInitContainerRuntime="3.789166895s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:43:50.144435859 +0000 UTC m=+2655.488837016" watchObservedRunningTime="2026-08-04 18:43:50.154793124 +0000 UTC m=+2655.499194281"
Aug 04 18:43:51.123948 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:51.123905    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:43:52.128214 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:52.128166    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:43:57.134606 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:57.134576    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:43:57.135108 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:57.135079    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:43:58.877952 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:43:58.877904    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:44:07.135991 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:07.135944    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:44:08.877697 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:08.877654    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:44:17.135528 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:17.135482    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:44:18.877773 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:18.877738    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:44:27.135549 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:27.135443    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:44:28.527253 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:28.527212    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"]
Aug 04 18:44:28.527771 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:28.527544    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" containerID="cri-o://99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24" gracePeriod=30
Aug 04 18:44:28.527771 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:28.527653    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kube-rbac-proxy" containerID="cri-o://c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030" gracePeriod=30
Aug 04 18:44:28.545090 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:28.545030    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:44:28.872573 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:28.872477    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Aug 04 18:44:28.877945 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:28.877903    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:44:29.279816 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:29.279782    2581 generic.go:356] "Generic (PLEG): container finished" podID="fed8a32a-215b-44fc-864b-86008b814ab6" containerID="c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030" exitCode=2
Aug 04 18:44:29.279989 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:29.279859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" event={"ID":"fed8a32a-215b-44fc-864b-86008b814ab6","Type":"ContainerDied","Data":"c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030"}
Aug 04 18:44:33.706006 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:33.705950    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:44:33.832691 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:33.832651    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:44:33.872751 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:33.872704    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Aug 04 18:44:34.783585 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.783554    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:44:34.792316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.792279    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr6hr\" (UniqueName: \"kubernetes.io/projected/fed8a32a-215b-44fc-864b-86008b814ab6-kube-api-access-tr6hr\") pod \"fed8a32a-215b-44fc-864b-86008b814ab6\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") "
Aug 04 18:44:34.792504 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.792336    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-cabundle-cert\") pod \"fed8a32a-215b-44fc-864b-86008b814ab6\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") "
Aug 04 18:44:34.792504 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.792366    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-kube-rbac-proxy-sar-config\") pod \"fed8a32a-215b-44fc-864b-86008b814ab6\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") "
Aug 04 18:44:34.792504 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.792407    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fed8a32a-215b-44fc-864b-86008b814ab6-kserve-provision-location\") pod \"fed8a32a-215b-44fc-864b-86008b814ab6\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") "
Aug 04 18:44:34.792504 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.792442    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fed8a32a-215b-44fc-864b-86008b814ab6-proxy-tls\") pod \"fed8a32a-215b-44fc-864b-86008b814ab6\" (UID: \"fed8a32a-215b-44fc-864b-86008b814ab6\") "
Aug 04 18:44:34.792727 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.792704    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fed8a32a-215b-44fc-864b-86008b814ab6" (UID: "fed8a32a-215b-44fc-864b-86008b814ab6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:44:34.792792 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.792734    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fed8a32a-215b-44fc-864b-86008b814ab6" (UID: "fed8a32a-215b-44fc-864b-86008b814ab6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:44:34.792792 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.792761    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed8a32a-215b-44fc-864b-86008b814ab6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fed8a32a-215b-44fc-864b-86008b814ab6" (UID: "fed8a32a-215b-44fc-864b-86008b814ab6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:44:34.794718 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.794682    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed8a32a-215b-44fc-864b-86008b814ab6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fed8a32a-215b-44fc-864b-86008b814ab6" (UID: "fed8a32a-215b-44fc-864b-86008b814ab6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:44:34.794835 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.794745    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed8a32a-215b-44fc-864b-86008b814ab6-kube-api-access-tr6hr" (OuterVolumeSpecName: "kube-api-access-tr6hr") pod "fed8a32a-215b-44fc-864b-86008b814ab6" (UID: "fed8a32a-215b-44fc-864b-86008b814ab6"). InnerVolumeSpecName "kube-api-access-tr6hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:44:34.893466 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.893382    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fed8a32a-215b-44fc-864b-86008b814ab6-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:44:34.893466 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.893414    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fed8a32a-215b-44fc-864b-86008b814ab6-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:44:34.893466 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.893426    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr6hr\" (UniqueName: \"kubernetes.io/projected/fed8a32a-215b-44fc-864b-86008b814ab6-kube-api-access-tr6hr\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:44:34.893466 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.893434    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:44:34.893466 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:34.893443    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fed8a32a-215b-44fc-864b-86008b814ab6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:44:35.312444 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.312404    2581 generic.go:356] "Generic (PLEG): container finished" podID="fed8a32a-215b-44fc-864b-86008b814ab6" containerID="99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24" exitCode=0
Aug 04 18:44:35.312658 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.312479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" event={"ID":"fed8a32a-215b-44fc-864b-86008b814ab6","Type":"ContainerDied","Data":"99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24"}
Aug 04 18:44:35.312658 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.312502    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"
Aug 04 18:44:35.312658 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.312559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z" event={"ID":"fed8a32a-215b-44fc-864b-86008b814ab6","Type":"ContainerDied","Data":"9a27f62fb87bd433479c229b282a82c2e4ef8ea2a0f329df5340fce1c55271dd"}
Aug 04 18:44:35.312658 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.312589    2581 scope.go:117] "RemoveContainer" containerID="c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030"
Aug 04 18:44:35.323188 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.323107    2581 scope.go:117] "RemoveContainer" containerID="99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24"
Aug 04 18:44:35.329349 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.329319    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:44:35.329587 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.329571    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:44:35.333331 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.333304    2581 scope.go:117] "RemoveContainer" containerID="c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905"
Aug 04 18:44:35.334594 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.334572    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"]
Aug 04 18:44:35.339323 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.339292    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-6sv2z"]
Aug 04 18:44:35.339540 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.339519    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:44:35.339802 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.339783    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:44:35.343368 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.343346    2581 scope.go:117] "RemoveContainer" containerID="c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030"
Aug 04 18:44:35.343604 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:44:35.343582    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030\": container with ID starting with c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030 not found: ID does not exist" containerID="c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030"
Aug 04 18:44:35.343647 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.343614    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030"} err="failed to get container status \"c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030\": rpc error: code = NotFound desc = could not find container \"c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030\": container with ID starting with c46ae9d59c284f9627f7870f77d9247d3e83be698cc57548acc34e8a698e0030 not found: ID does not exist"
Aug 04 18:44:35.343647 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.343635    2581 scope.go:117] "RemoveContainer" containerID="99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24"
Aug 04 18:44:35.343867 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:44:35.343850    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24\": container with ID starting with 99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24 not found: ID does not exist" containerID="99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24"
Aug 04 18:44:35.343909 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.343869    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24"} err="failed to get container status \"99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24\": rpc error: code = NotFound desc = could not find container \"99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24\": container with ID starting with 99984646cc9385898849cd7129d6ecf46a3fc3c1f584d86098a5403b3dbc1a24 not found: ID does not exist"
Aug 04 18:44:35.343909 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.343881    2581 scope.go:117] "RemoveContainer" containerID="c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905"
Aug 04 18:44:35.344092 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:44:35.344071    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905\": container with ID starting with c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905 not found: ID does not exist" containerID="c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905"
Aug 04 18:44:35.344140 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:35.344097    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905"} err="failed to get container status \"c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905\": rpc error: code = NotFound desc = could not find container \"c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905\": container with ID starting with c054f9b1d1c45c69e93a28e7742436294d637befb1bbf3cb7dcbff8bb9b36905 not found: ID does not exist"
Aug 04 18:44:37.136187 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:37.136138    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:44:37.214388 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:37.214350    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" path="/var/lib/kubelet/pods/fed8a32a-215b-44fc-864b-86008b814ab6/volumes"
Aug 04 18:44:47.135341 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:47.135292    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:44:57.136678 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:44:57.136638    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:45:04.087033 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:04.086999    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"]
Aug 04 18:45:04.087571 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:04.087283    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container" containerID="cri-o://9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e" gracePeriod=30
Aug 04 18:45:04.087571 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:04.087341    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kube-rbac-proxy" containerID="cri-o://ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534" gracePeriod=30
Aug 04 18:45:04.226622 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:04.226587    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:04.429926 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:04.429834    2581 generic.go:356] "Generic (PLEG): container finished" podID="cbc3819e-397f-490e-8125-adb8b62445fc" containerID="ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534" exitCode=2
Aug 04 18:45:04.429926 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:04.429871    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" event={"ID":"cbc3819e-397f-490e-8125-adb8b62445fc","Type":"ContainerDied","Data":"ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534"}
Aug 04 18:45:07.129031 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:07.128984    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Aug 04 18:45:07.135639 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:07.135595    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 04 18:45:08.036939 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.036911    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:45:08.157636 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.157515    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc3819e-397f-490e-8125-adb8b62445fc-kserve-provision-location\") pod \"cbc3819e-397f-490e-8125-adb8b62445fc\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") "
Aug 04 18:45:08.157636 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.157579    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcz49\" (UniqueName: \"kubernetes.io/projected/cbc3819e-397f-490e-8125-adb8b62445fc-kube-api-access-qcz49\") pod \"cbc3819e-397f-490e-8125-adb8b62445fc\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") "
Aug 04 18:45:08.157636 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.157602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc3819e-397f-490e-8125-adb8b62445fc-proxy-tls\") pod \"cbc3819e-397f-490e-8125-adb8b62445fc\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") "
Aug 04 18:45:08.157636 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.157630    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-kube-rbac-proxy-sar-config\") pod \"cbc3819e-397f-490e-8125-adb8b62445fc\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") "
Aug 04 18:45:08.158247 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.157700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-cabundle-cert\") pod \"cbc3819e-397f-490e-8125-adb8b62445fc\" (UID: \"cbc3819e-397f-490e-8125-adb8b62445fc\") "
Aug 04 18:45:08.158247 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.158007    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc3819e-397f-490e-8125-adb8b62445fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cbc3819e-397f-490e-8125-adb8b62445fc" (UID: "cbc3819e-397f-490e-8125-adb8b62445fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:08.158247 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.158104    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cbc3819e-397f-490e-8125-adb8b62445fc" (UID: "cbc3819e-397f-490e-8125-adb8b62445fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:45:08.158247 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.158102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cbc3819e-397f-490e-8125-adb8b62445fc" (UID: "cbc3819e-397f-490e-8125-adb8b62445fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:45:08.159919 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.159896    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc3819e-397f-490e-8125-adb8b62445fc-kube-api-access-qcz49" (OuterVolumeSpecName: "kube-api-access-qcz49") pod "cbc3819e-397f-490e-8125-adb8b62445fc" (UID: "cbc3819e-397f-490e-8125-adb8b62445fc"). InnerVolumeSpecName "kube-api-access-qcz49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:45:08.159919 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.159909    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc3819e-397f-490e-8125-adb8b62445fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cbc3819e-397f-490e-8125-adb8b62445fc" (UID: "cbc3819e-397f-490e-8125-adb8b62445fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:45:08.258889 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.258843    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:45:08.258889 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.258878    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc3819e-397f-490e-8125-adb8b62445fc-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:45:08.258889 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.258893    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcz49\" (UniqueName: \"kubernetes.io/projected/cbc3819e-397f-490e-8125-adb8b62445fc-kube-api-access-qcz49\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:45:08.259174 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.258905    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc3819e-397f-490e-8125-adb8b62445fc-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:45:08.259174 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.258918    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc3819e-397f-490e-8125-adb8b62445fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:45:08.449786 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.449693    2581 generic.go:356] "Generic (PLEG): container finished" podID="cbc3819e-397f-490e-8125-adb8b62445fc" containerID="9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e" exitCode=0
Aug 04 18:45:08.449786 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.449750    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"
Aug 04 18:45:08.450026 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.449787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" event={"ID":"cbc3819e-397f-490e-8125-adb8b62445fc","Type":"ContainerDied","Data":"9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e"}
Aug 04 18:45:08.450026 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.449863    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf" event={"ID":"cbc3819e-397f-490e-8125-adb8b62445fc","Type":"ContainerDied","Data":"ad3269218c74a41748a9cc44a5fd883ab1082fd34607255a978e81b38654cde5"}
Aug 04 18:45:08.450026 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.449902    2581 scope.go:117] "RemoveContainer" containerID="ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534"
Aug 04 18:45:08.459114 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.459026    2581 scope.go:117] "RemoveContainer" containerID="9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e"
Aug 04 18:45:08.467339 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.467316    2581 scope.go:117] "RemoveContainer" containerID="dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181"
Aug 04 18:45:08.473826 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.473791    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"]
Aug 04 18:45:08.475976 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.475949    2581 scope.go:117] "RemoveContainer" containerID="ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534"
Aug 04 18:45:08.476321 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:45:08.476298    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534\": container with ID starting with ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534 not found: ID does not exist" containerID="ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534"
Aug 04 18:45:08.476383 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.476330    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534"} err="failed to get container status \"ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534\": rpc error: code = NotFound desc = could not find container \"ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534\": container with ID starting with ffec77cf7ce947b952af918aa54a8a16e78ee8b01636498fc3ba7a7021169534 not found: ID does not exist"
Aug 04 18:45:08.476383 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.476348    2581 scope.go:117] "RemoveContainer" containerID="9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e"
Aug 04 18:45:08.476868 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:45:08.476825    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e\": container with ID starting with 9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e not found: ID does not exist" containerID="9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e"
Aug 04 18:45:08.476986 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.476863    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e"} err="failed to get container status \"9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e\": rpc error: code = NotFound desc = could not find container \"9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e\": container with ID starting with 9a6e684e6dd030cc39c04a79ecaf7c9852a5bfc02e7235c18762df7dc019f87e not found: ID does not exist"
Aug 04 18:45:08.476986 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.476887    2581 scope.go:117] "RemoveContainer" containerID="dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181"
Aug 04 18:45:08.477382 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:45:08.477353    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181\": container with ID starting with dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181 not found: ID does not exist" containerID="dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181"
Aug 04 18:45:08.477473 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.477382    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181"} err="failed to get container status \"dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181\": rpc error: code = NotFound desc = could not find container \"dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181\": container with ID starting with dd3e34bac1c3dd291d198fea233f8613585836b4cd4cad3053a167b99c81b181 not found: ID does not exist"
Aug 04 18:45:08.478710 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:08.478689    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-c5hhf"]
Aug 04 18:45:09.213540 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:09.213503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:09.215381 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:09.215353    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" path="/var/lib/kubelet/pods/cbc3819e-397f-490e-8125-adb8b62445fc/volumes"
Aug 04 18:45:09.445599 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:09.445562    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:23.839071 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:23.839013    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:28.944815 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.944779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:28.977384 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977347    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"]
Aug 04 18:45:28.977711 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977695    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container"
Aug 04 18:45:28.977806 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977714    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container"
Aug 04 18:45:28.977806 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977730    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="storage-initializer"
Aug 04 18:45:28.977806 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977738    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="storage-initializer"
Aug 04 18:45:28.977806 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977758    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="storage-initializer"
Aug 04 18:45:28.977806 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977766    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="storage-initializer"
Aug 04 18:45:28.977806 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977784    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kube-rbac-proxy"
Aug 04 18:45:28.977806 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977793    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kube-rbac-proxy"
Aug 04 18:45:28.977806 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977804    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kube-rbac-proxy"
Aug 04 18:45:28.978230 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977812    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kube-rbac-proxy"
Aug 04 18:45:28.978230 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977820    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container"
Aug 04 18:45:28.978230 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977827    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container"
Aug 04 18:45:28.978230 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977925    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kserve-container"
Aug 04 18:45:28.978230 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977938    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kserve-container"
Aug 04 18:45:28.978230 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977951    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc3819e-397f-490e-8125-adb8b62445fc" containerName="kube-rbac-proxy"
Aug 04 18:45:28.978230 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.977960    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fed8a32a-215b-44fc-864b-86008b814ab6" containerName="kube-rbac-proxy"
Aug 04 18:45:28.980439 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.980416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:28.982932 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.982908    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:45:28.983133 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.983114    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Aug 04 18:45:28.983210 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.983131    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:45:28.984028 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.983932    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:45:28.984028 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.983963    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:45:28.984028 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.984002    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Aug 04 18:45:28.984028 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.984004    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:45:28.991501 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:28.991475    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"]
Aug 04 18:45:29.134988 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.134942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whkq\" (UniqueName: \"kubernetes.io/projected/701d4876-70ca-44b3-9551-2566b3898dfe-kube-api-access-4whkq\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.135201 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.135009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/701d4876-70ca-44b3-9551-2566b3898dfe-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.135201 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.135073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/701d4876-70ca-44b3-9551-2566b3898dfe-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.135201 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.135101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.135201 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.135137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.219285 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.219165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:29.235678 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.235631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whkq\" (UniqueName: \"kubernetes.io/projected/701d4876-70ca-44b3-9551-2566b3898dfe-kube-api-access-4whkq\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.235873 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.235752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/701d4876-70ca-44b3-9551-2566b3898dfe-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.235873 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.235797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/701d4876-70ca-44b3-9551-2566b3898dfe-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.236009 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.235979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.236118 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.236057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.236248 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.236202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/701d4876-70ca-44b3-9551-2566b3898dfe-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.236676 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.236653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.236766 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.236684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.238565 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.238543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/701d4876-70ca-44b3-9551-2566b3898dfe-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.245759 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.245727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whkq\" (UniqueName: \"kubernetes.io/projected/701d4876-70ca-44b3-9551-2566b3898dfe-kube-api-access-4whkq\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.292635 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.292595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:29.430024 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.429998    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"]
Aug 04 18:45:29.432739 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:45:29.432704    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701d4876_70ca_44b3_9551_2566b3898dfe.slice/crio-133d09002feab99b1f2461681f71f8ff1e3f5a604a961fe40062134f9c00fbb2 WatchSource:0}: Error finding container 133d09002feab99b1f2461681f71f8ff1e3f5a604a961fe40062134f9c00fbb2: Status 404 returned error can't find the container with id 133d09002feab99b1f2461681f71f8ff1e3f5a604a961fe40062134f9c00fbb2
Aug 04 18:45:29.528822 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.528776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" event={"ID":"701d4876-70ca-44b3-9551-2566b3898dfe","Type":"ContainerStarted","Data":"ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973"}
Aug 04 18:45:29.529017 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:29.528835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" event={"ID":"701d4876-70ca-44b3-9551-2566b3898dfe","Type":"ContainerStarted","Data":"133d09002feab99b1f2461681f71f8ff1e3f5a604a961fe40062134f9c00fbb2"}
Aug 04 18:45:33.545832 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:33.545799    2581 generic.go:356] "Generic (PLEG): container finished" podID="701d4876-70ca-44b3-9551-2566b3898dfe" containerID="ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973" exitCode=0
Aug 04 18:45:33.546295 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:33.545849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" event={"ID":"701d4876-70ca-44b3-9551-2566b3898dfe","Type":"ContainerDied","Data":"ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973"}
Aug 04 18:45:34.553522 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:34.553474    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" event={"ID":"701d4876-70ca-44b3-9551-2566b3898dfe","Type":"ContainerStarted","Data":"39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c"}
Aug 04 18:45:34.554032 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:34.553554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" event={"ID":"701d4876-70ca-44b3-9551-2566b3898dfe","Type":"ContainerStarted","Data":"c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b"}
Aug 04 18:45:34.554032 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:34.553770    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:34.574849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:34.574785    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" podStartSLOduration=2.895561155 podStartE2EDuration="6.574768219s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.679207064s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:34.57374429 +0000 UTC m=+2759.918145448" watchObservedRunningTime="2026-08-04 18:45:34.574768219 +0000 UTC m=+2759.919169375"
Aug 04 18:45:35.558497 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:35.558461    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:35.559850 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:35.559821    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 04 18:45:36.562124 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:36.562067    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Aug 04 18:45:41.566175 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:41.566145    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:41.567231 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:41.567207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:49.048418 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.048380    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"]
Aug 04 18:45:49.048907 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.048714    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="kserve-container" containerID="cri-o://c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b" gracePeriod=30
Aug 04 18:45:49.048907 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.048784    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="kube-rbac-proxy" containerID="cri-o://39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c" gracePeriod=30
Aug 04 18:45:49.087006 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.086968    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:49.614211 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.614181    2581 generic.go:356] "Generic (PLEG): container finished" podID="701d4876-70ca-44b3-9551-2566b3898dfe" containerID="39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c" exitCode=2
Aug 04 18:45:49.614330 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.614239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" event={"ID":"701d4876-70ca-44b3-9551-2566b3898dfe","Type":"ContainerDied","Data":"39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c"}
Aug 04 18:45:49.790458 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.790431    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:49.813742 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.813711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/701d4876-70ca-44b3-9551-2566b3898dfe-kserve-provision-location\") pod \"701d4876-70ca-44b3-9551-2566b3898dfe\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") "
Aug 04 18:45:49.813926 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.813772    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4whkq\" (UniqueName: \"kubernetes.io/projected/701d4876-70ca-44b3-9551-2566b3898dfe-kube-api-access-4whkq\") pod \"701d4876-70ca-44b3-9551-2566b3898dfe\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") "
Aug 04 18:45:49.813926 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.813815    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-cabundle-cert\") pod \"701d4876-70ca-44b3-9551-2566b3898dfe\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") "
Aug 04 18:45:49.813926 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.813859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/701d4876-70ca-44b3-9551-2566b3898dfe-proxy-tls\") pod \"701d4876-70ca-44b3-9551-2566b3898dfe\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") "
Aug 04 18:45:49.813926 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.813901    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-kube-rbac-proxy-sar-config\") pod \"701d4876-70ca-44b3-9551-2566b3898dfe\" (UID: \"701d4876-70ca-44b3-9551-2566b3898dfe\") "
Aug 04 18:45:49.814167 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.814142    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701d4876-70ca-44b3-9551-2566b3898dfe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "701d4876-70ca-44b3-9551-2566b3898dfe" (UID: "701d4876-70ca-44b3-9551-2566b3898dfe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:49.814503 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.814475    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "701d4876-70ca-44b3-9551-2566b3898dfe" (UID: "701d4876-70ca-44b3-9551-2566b3898dfe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:45:49.814626 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.814526    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "701d4876-70ca-44b3-9551-2566b3898dfe" (UID: "701d4876-70ca-44b3-9551-2566b3898dfe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:45:49.816630 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.816599    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701d4876-70ca-44b3-9551-2566b3898dfe-kube-api-access-4whkq" (OuterVolumeSpecName: "kube-api-access-4whkq") pod "701d4876-70ca-44b3-9551-2566b3898dfe" (UID: "701d4876-70ca-44b3-9551-2566b3898dfe"). InnerVolumeSpecName "kube-api-access-4whkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:45:49.816899 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.816870    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/701d4876-70ca-44b3-9551-2566b3898dfe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "701d4876-70ca-44b3-9551-2566b3898dfe" (UID: "701d4876-70ca-44b3-9551-2566b3898dfe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:45:49.914672 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.914579    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/701d4876-70ca-44b3-9551-2566b3898dfe-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:45:49.914672 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.914608    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whkq\" (UniqueName: \"kubernetes.io/projected/701d4876-70ca-44b3-9551-2566b3898dfe-kube-api-access-4whkq\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:45:49.914672 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.914618    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:45:49.914672 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.914627    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/701d4876-70ca-44b3-9551-2566b3898dfe-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:45:49.914672 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:49.914635    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/701d4876-70ca-44b3-9551-2566b3898dfe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:45:50.620254 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.620211    2581 generic.go:356] "Generic (PLEG): container finished" podID="701d4876-70ca-44b3-9551-2566b3898dfe" containerID="c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b" exitCode=0
Aug 04 18:45:50.620722 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.620267    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" event={"ID":"701d4876-70ca-44b3-9551-2566b3898dfe","Type":"ContainerDied","Data":"c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b"}
Aug 04 18:45:50.620722 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.620325    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"
Aug 04 18:45:50.620722 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.620345    2581 scope.go:117] "RemoveContainer" containerID="39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c"
Aug 04 18:45:50.620722 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.620331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52" event={"ID":"701d4876-70ca-44b3-9551-2566b3898dfe","Type":"ContainerDied","Data":"133d09002feab99b1f2461681f71f8ff1e3f5a604a961fe40062134f9c00fbb2"}
Aug 04 18:45:50.629630 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.629612    2581 scope.go:117] "RemoveContainer" containerID="c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b"
Aug 04 18:45:50.637775 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.637753    2581 scope.go:117] "RemoveContainer" containerID="ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973"
Aug 04 18:45:50.645216 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.645186    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"]
Aug 04 18:45:50.648510 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.647170    2581 scope.go:117] "RemoveContainer" containerID="39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c"
Aug 04 18:45:50.649218 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:45:50.649192    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c\": container with ID starting with 39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c not found: ID does not exist" containerID="39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c"
Aug 04 18:45:50.649343 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.649232    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c"} err="failed to get container status \"39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c\": rpc error: code = NotFound desc = could not find container \"39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c\": container with ID starting with 39b71be4b3e4a0c78fceee0e3899b3f760dfa9ebd32a010882a148fb6f6fff8c not found: ID does not exist"
Aug 04 18:45:50.649343 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.649253    2581 scope.go:117] "RemoveContainer" containerID="c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b"
Aug 04 18:45:50.649732 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:45:50.649700    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b\": container with ID starting with c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b not found: ID does not exist" containerID="c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b"
Aug 04 18:45:50.649829 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.649736    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b"} err="failed to get container status \"c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b\": rpc error: code = NotFound desc = could not find container \"c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b\": container with ID starting with c894187622c65c92cec28b5ca2338a81448a31b6a2dbb852526e9e8eadc92d5b not found: ID does not exist"
Aug 04 18:45:50.649829 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.649755    2581 scope.go:117] "RemoveContainer" containerID="ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973"
Aug 04 18:45:50.650096 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:45:50.650075    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973\": container with ID starting with ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973 not found: ID does not exist" containerID="ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973"
Aug 04 18:45:50.650167 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.650105    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973"} err="failed to get container status \"ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973\": rpc error: code = NotFound desc = could not find container \"ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973\": container with ID starting with ac6aa4d32e30b384a50c0f95a10aa7788f2c748eba6f8ed97d00a6fe64cfd973 not found: ID does not exist"
Aug 04 18:45:50.650453 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:50.650432    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n5t52"]
Aug 04 18:45:51.214744 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:51.214704    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" path="/var/lib/kubelet/pods/701d4876-70ca-44b3-9551-2566b3898dfe/volumes"
Aug 04 18:45:54.213335 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:54.213299    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:54.718132 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:54.718091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:54.782766 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:45:54.782729    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:09.352599 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:09.352557    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:14.483075 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.483027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:14.504758 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.504721    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"]
Aug 04 18:46:14.505073 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.505038    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="kube-rbac-proxy"
Aug 04 18:46:14.505073 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.505074    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="kube-rbac-proxy"
Aug 04 18:46:14.505224 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.505097    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="storage-initializer"
Aug 04 18:46:14.505224 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.505103    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="storage-initializer"
Aug 04 18:46:14.505224 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.505111    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="kserve-container"
Aug 04 18:46:14.505224 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.505116    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="kserve-container"
Aug 04 18:46:14.505224 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.505166    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="kserve-container"
Aug 04 18:46:14.505224 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.505176    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="701d4876-70ca-44b3-9551-2566b3898dfe" containerName="kube-rbac-proxy"
Aug 04 18:46:14.507538 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.507520    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.510385 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.510359    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:46:14.510584 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.510359    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:46:14.510679 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.510363    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Aug 04 18:46:14.510679 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.510376    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Aug 04 18:46:14.510679 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.510468    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:46:14.511226 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.511205    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:46:14.511341 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.511262    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:46:14.518782 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.518759    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"]
Aug 04 18:46:14.604031 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.603998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64qdp\" (UniqueName: \"kubernetes.io/projected/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-api-access-64qdp\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.604248 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.604074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.604248 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.604143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.604248 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.604223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.604365 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.604284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.701382 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.701345    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:14.704864 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.704819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.704864 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.704864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.705095 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.704899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.705095 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.704947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.705215 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.705104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64qdp\" (UniqueName: \"kubernetes.io/projected/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-api-access-64qdp\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.705414 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.705394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.705828 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.705806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.705964 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.705942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.707885 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.707864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.715015 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.714986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64qdp\" (UniqueName: \"kubernetes.io/projected/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-api-access-64qdp\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pzghz\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.819090 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.819028    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:14.982608 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:14.982571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"]
Aug 04 18:46:14.986203 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:46:14.986168    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e60f20_8b9c_453c_b3ff_abba6fd4c654.slice/crio-35585ecea73d7beed8d9d96da02f3e2537c3017a02e7982a45366dbb280d79d1 WatchSource:0}: Error finding container 35585ecea73d7beed8d9d96da02f3e2537c3017a02e7982a45366dbb280d79d1: Status 404 returned error can't find the container with id 35585ecea73d7beed8d9d96da02f3e2537c3017a02e7982a45366dbb280d79d1
Aug 04 18:46:15.715050 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:15.715012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" event={"ID":"d5e60f20-8b9c-453c-b3ff-abba6fd4c654","Type":"ContainerStarted","Data":"88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0"}
Aug 04 18:46:15.715409 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:15.715098    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" event={"ID":"d5e60f20-8b9c-453c-b3ff-abba6fd4c654","Type":"ContainerStarted","Data":"35585ecea73d7beed8d9d96da02f3e2537c3017a02e7982a45366dbb280d79d1"}
Aug 04 18:46:19.731866 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:19.731833    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerID="88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0" exitCode=0
Aug 04 18:46:19.732285 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:19.731907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" event={"ID":"d5e60f20-8b9c-453c-b3ff-abba6fd4c654","Type":"ContainerDied","Data":"88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0"}
Aug 04 18:46:20.737900 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:20.737865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" event={"ID":"d5e60f20-8b9c-453c-b3ff-abba6fd4c654","Type":"ContainerStarted","Data":"f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6"}
Aug 04 18:46:20.738293 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:20.737926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" event={"ID":"d5e60f20-8b9c-453c-b3ff-abba6fd4c654","Type":"ContainerStarted","Data":"6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807"}
Aug 04 18:46:20.738293 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:20.738204    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:20.762637 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:20.762583    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podStartSLOduration=3.022104945 podStartE2EDuration="6.762567231s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.740462286s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:46:20.761633981 +0000 UTC m=+2806.106035137" watchObservedRunningTime="2026-08-04 18:46:20.762567231 +0000 UTC m=+2806.106968390"
Aug 04 18:46:21.741614 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:21.741582    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:21.743017 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:21.742993    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:46:22.745429 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:22.745388    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:46:27.750548 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:27.750514    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:46:27.751208 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:27.751178    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:46:37.751936 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:37.751876    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:46:44.357905 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:44.357869    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:47.751157 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:47.751116    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:46:49.516527 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:49.516482    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:49.797002 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:49.796972    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:46:57.751732 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:46:57.751680    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:47:07.751386 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:07.751347    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:47:17.751216 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:17.751171    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:47:27.752315 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:27.752227    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:47:34.651882 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:34.651846    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"]
Aug 04 18:47:34.652294 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:34.652178    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container" containerID="cri-o://6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807" gracePeriod=30
Aug 04 18:47:34.652294 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:34.652197    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kube-rbac-proxy" containerID="cri-o://f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6" gracePeriod=30
Aug 04 18:47:34.682566 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:34.682535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:47:35.025865 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:35.025832    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerID="f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6" exitCode=2
Aug 04 18:47:35.026038 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:35.025870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" event={"ID":"d5e60f20-8b9c-453c-b3ff-abba6fd4c654","Type":"ContainerDied","Data":"f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6"}
Aug 04 18:47:37.745990 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:37.745947    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Aug 04 18:47:37.751432 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:37.751396    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Aug 04 18:47:38.597363 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.597334    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:47:38.668003 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.667912    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-rbac-proxy-sar-config\") pod \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") "
Aug 04 18:47:38.668003 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.667957    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-proxy-tls\") pod \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") "
Aug 04 18:47:38.668003 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.667989    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kserve-provision-location\") pod \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") "
Aug 04 18:47:38.668329 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.668064    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-cabundle-cert\") pod \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") "
Aug 04 18:47:38.668329 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.668110    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64qdp\" (UniqueName: \"kubernetes.io/projected/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-api-access-64qdp\") pod \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\" (UID: \"d5e60f20-8b9c-453c-b3ff-abba6fd4c654\") "
Aug 04 18:47:38.668436 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.668368    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5e60f20-8b9c-453c-b3ff-abba6fd4c654" (UID: "d5e60f20-8b9c-453c-b3ff-abba6fd4c654"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:47:38.668436 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.668380    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5e60f20-8b9c-453c-b3ff-abba6fd4c654" (UID: "d5e60f20-8b9c-453c-b3ff-abba6fd4c654"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:47:38.668436 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.668419    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5e60f20-8b9c-453c-b3ff-abba6fd4c654" (UID: "d5e60f20-8b9c-453c-b3ff-abba6fd4c654"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:47:38.670351 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.670323    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5e60f20-8b9c-453c-b3ff-abba6fd4c654" (UID: "d5e60f20-8b9c-453c-b3ff-abba6fd4c654"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:47:38.670525 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.670509    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-api-access-64qdp" (OuterVolumeSpecName: "kube-api-access-64qdp") pod "d5e60f20-8b9c-453c-b3ff-abba6fd4c654" (UID: "d5e60f20-8b9c-453c-b3ff-abba6fd4c654"). InnerVolumeSpecName "kube-api-access-64qdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:47:38.769170 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.769131    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:47:38.769170 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.769163    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:47:38.769170 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.769174    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:47:38.769170 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.769184    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:47:38.769649 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:38.769193    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64qdp\" (UniqueName: \"kubernetes.io/projected/d5e60f20-8b9c-453c-b3ff-abba6fd4c654-kube-api-access-64qdp\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:47:39.043059 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.043015    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerID="6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807" exitCode=0
Aug 04 18:47:39.043225 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.043094    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"
Aug 04 18:47:39.043225 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.043089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" event={"ID":"d5e60f20-8b9c-453c-b3ff-abba6fd4c654","Type":"ContainerDied","Data":"6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807"}
Aug 04 18:47:39.043298 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.043229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz" event={"ID":"d5e60f20-8b9c-453c-b3ff-abba6fd4c654","Type":"ContainerDied","Data":"35585ecea73d7beed8d9d96da02f3e2537c3017a02e7982a45366dbb280d79d1"}
Aug 04 18:47:39.043298 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.043257    2581 scope.go:117] "RemoveContainer" containerID="f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6"
Aug 04 18:47:39.052272 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.052251    2581 scope.go:117] "RemoveContainer" containerID="6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807"
Aug 04 18:47:39.060779 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.060759    2581 scope.go:117] "RemoveContainer" containerID="88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0"
Aug 04 18:47:39.069128 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.069103    2581 scope.go:117] "RemoveContainer" containerID="f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6"
Aug 04 18:47:39.069461 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:47:39.069441    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6\": container with ID starting with f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6 not found: ID does not exist" containerID="f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6"
Aug 04 18:47:39.069510 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.069473    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6"} err="failed to get container status \"f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6\": rpc error: code = NotFound desc = could not find container \"f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6\": container with ID starting with f4c333e8aad1ae7fa7675c8458215dc0862bd33f6e8ea5ccbac5402e31582ef6 not found: ID does not exist"
Aug 04 18:47:39.069510 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.069493    2581 scope.go:117] "RemoveContainer" containerID="6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807"
Aug 04 18:47:39.069787 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:47:39.069762    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807\": container with ID starting with 6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807 not found: ID does not exist" containerID="6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807"
Aug 04 18:47:39.069877 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.069800    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807"} err="failed to get container status \"6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807\": rpc error: code = NotFound desc = could not find container \"6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807\": container with ID starting with 6d7484b3bc79309bd0bcb34387323638f245e2335e2dcf2749b18ce9af37f807 not found: ID does not exist"
Aug 04 18:47:39.069877 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.069823    2581 scope.go:117] "RemoveContainer" containerID="88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0"
Aug 04 18:47:39.070133 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:47:39.070106    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0\": container with ID starting with 88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0 not found: ID does not exist" containerID="88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0"
Aug 04 18:47:39.070183 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.070130    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0"} err="failed to get container status \"88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0\": rpc error: code = NotFound desc = could not find container \"88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0\": container with ID starting with 88abcf696f9b6efe9e0e0aa75df2ad1e6d8e68a2bfdeddec4bf39ee0c8d6baf0 not found: ID does not exist"
Aug 04 18:47:39.076546 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.076514    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"]
Aug 04 18:47:39.087501 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.087469    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pzghz"]
Aug 04 18:47:39.214212 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.214175    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" path="/var/lib/kubelet/pods/d5e60f20-8b9c-453c-b3ff-abba6fd4c654/volumes"
Aug 04 18:47:39.844239 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.844209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:47:39.901510 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.901474    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"]
Aug 04 18:47:39.901772 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.901760    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container"
Aug 04 18:47:39.901831 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.901774    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container"
Aug 04 18:47:39.901831 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.901784    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="storage-initializer"
Aug 04 18:47:39.901831 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.901790    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="storage-initializer"
Aug 04 18:47:39.901831 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.901803    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kube-rbac-proxy"
Aug 04 18:47:39.901831 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.901808    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kube-rbac-proxy"
Aug 04 18:47:39.901988 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.901859    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kube-rbac-proxy"
Aug 04 18:47:39.901988 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.901868    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e60f20-8b9c-453c-b3ff-abba6fd4c654" containerName="kserve-container"
Aug 04 18:47:39.905341 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.905317    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:39.909674 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.909651    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Aug 04 18:47:39.909836 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.909651    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:47:39.909836 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.909651    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Aug 04 18:47:39.909836 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.909661    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Aug 04 18:47:39.910022 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.909919    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:47:39.910022 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.909944    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:47:39.917699 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.917674    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"]
Aug 04 18:47:39.978879 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.978833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:39.979083 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.978887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4h6\" (UniqueName: \"kubernetes.io/projected/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-api-access-2m4h6\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:39.979083 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.978971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0aae61-4b05-4aa1-b966-8250488dbe1b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:39.979083 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:39.979064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:40.048518 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:40.048486    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:47:40.080523 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:40.080489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:40.080714 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:40.080554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:40.080714 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:40.080582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4h6\" (UniqueName: \"kubernetes.io/projected/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-api-access-2m4h6\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:40.080714 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:40.080614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0aae61-4b05-4aa1-b966-8250488dbe1b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:40.081013 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:40.080991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:40.081384 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:40.081363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:40.083325 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:40.083301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0aae61-4b05-4aa1-b966-8250488dbe1b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:40.092125 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:40.092093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4h6\" (UniqueName: \"kubernetes.io/projected/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-api-access-2m4h6\") pod \"isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:40.215946 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:40.215854    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:40.363907 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:40.363876    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"]
Aug 04 18:47:40.365803 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:47:40.365772    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0aae61_4b05_4aa1_b966_8250488dbe1b.slice/crio-d37d3a0bf2ce3897c3f8b6924df81919c82095ef97e2bd57268bdef835666c0f WatchSource:0}: Error finding container d37d3a0bf2ce3897c3f8b6924df81919c82095ef97e2bd57268bdef835666c0f: Status 404 returned error can't find the container with id d37d3a0bf2ce3897c3f8b6924df81919c82095ef97e2bd57268bdef835666c0f
Aug 04 18:47:41.053936 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:41.053895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" event={"ID":"3a0aae61-4b05-4aa1-b966-8250488dbe1b","Type":"ContainerStarted","Data":"eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df"}
Aug 04 18:47:41.054336 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:41.053970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" event={"ID":"3a0aae61-4b05-4aa1-b966-8250488dbe1b","Type":"ContainerStarted","Data":"d37d3a0bf2ce3897c3f8b6924df81919c82095ef97e2bd57268bdef835666c0f"}
Aug 04 18:47:42.058825 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:42.058789    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerID="eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df" exitCode=0
Aug 04 18:47:42.059260 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:42.058854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" event={"ID":"3a0aae61-4b05-4aa1-b966-8250488dbe1b","Type":"ContainerDied","Data":"eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df"}
Aug 04 18:47:43.065811 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:43.065776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" event={"ID":"3a0aae61-4b05-4aa1-b966-8250488dbe1b","Type":"ContainerStarted","Data":"124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044"}
Aug 04 18:47:43.066252 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:43.065833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" event={"ID":"3a0aae61-4b05-4aa1-b966-8250488dbe1b","Type":"ContainerStarted","Data":"316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac"}
Aug 04 18:47:43.066252 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:43.065878    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:43.087343 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:43.087284    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podStartSLOduration=3.100933919 podStartE2EDuration="4.087265677s" totalImagesPullingTime="0s" totalInitContainerRuntime="986.331758ms" isStatefulPod=false podCreationTimestamp="2026-08-04 18:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:47:43.085573026 +0000 UTC m=+2888.429974186" watchObservedRunningTime="2026-08-04 18:47:43.087265677 +0000 UTC m=+2888.431666895"
Aug 04 18:47:44.070509 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:44.070475    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:44.071854 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:44.071825    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:47:45.074698 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:45.074646    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:47:50.080036 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:50.080008    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:47:50.080638 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:47:50.080608    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:48:00.080700 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:00.080658    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:48:09.644935 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:09.644900    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:10.081109 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:10.081066    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:48:14.874128 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:14.874091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:15.034703 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.034662    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"]
Aug 04 18:48:15.038512 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.038486    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.043804 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.043776    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Aug 04 18:48:15.044550 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.044518    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Aug 04 18:48:15.044674 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.044521    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:48:15.044674 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.044524    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:48:15.058491 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.058455    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"]
Aug 04 18:48:15.103951 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.103921    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:48:15.177340 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.177233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.177340 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.177325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.177548 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.177353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spq66\" (UniqueName: \"kubernetes.io/projected/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-api-access-spq66\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.177548 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.177382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d058778-1f7c-4d9a-8127-f077dbabff49-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.177548 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.177403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d058778-1f7c-4d9a-8127-f077dbabff49-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.278496 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.278455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.278690 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.278520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.278690 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.278546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spq66\" (UniqueName: \"kubernetes.io/projected/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-api-access-spq66\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.278690 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.278565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d058778-1f7c-4d9a-8127-f077dbabff49-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.278690 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.278582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d058778-1f7c-4d9a-8127-f077dbabff49-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.278990 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.278962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d058778-1f7c-4d9a-8127-f077dbabff49-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.279256 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.279236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.279327 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.279305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.281520 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.281495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d058778-1f7c-4d9a-8127-f077dbabff49-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.290493 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.290463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spq66\" (UniqueName: \"kubernetes.io/projected/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-api-access-spq66\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vztxp\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.349478 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.349427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:15.502855 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:15.502822    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"]
Aug 04 18:48:15.505233 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:48:15.505197    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d058778_1f7c_4d9a_8127_f077dbabff49.slice/crio-2d3dfa16eda1ae0a0e518f6ba15cac1e48eaae06082a3a91cccd09350addaf3c WatchSource:0}: Error finding container 2d3dfa16eda1ae0a0e518f6ba15cac1e48eaae06082a3a91cccd09350addaf3c: Status 404 returned error can't find the container with id 2d3dfa16eda1ae0a0e518f6ba15cac1e48eaae06082a3a91cccd09350addaf3c
Aug 04 18:48:16.212753 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:16.212712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" event={"ID":"9d058778-1f7c-4d9a-8127-f077dbabff49","Type":"ContainerStarted","Data":"e431dff4372c6e0c2164687be25ca1b5f338134c56dc538de3e4bd29c02fe027"}
Aug 04 18:48:16.213186 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:16.212772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" event={"ID":"9d058778-1f7c-4d9a-8127-f077dbabff49","Type":"ContainerStarted","Data":"2d3dfa16eda1ae0a0e518f6ba15cac1e48eaae06082a3a91cccd09350addaf3c"}
Aug 04 18:48:20.081538 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:20.081493    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:48:20.231540 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:20.231505    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerID="e431dff4372c6e0c2164687be25ca1b5f338134c56dc538de3e4bd29c02fe027" exitCode=0
Aug 04 18:48:20.231748 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:20.231583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" event={"ID":"9d058778-1f7c-4d9a-8127-f077dbabff49","Type":"ContainerDied","Data":"e431dff4372c6e0c2164687be25ca1b5f338134c56dc538de3e4bd29c02fe027"}
Aug 04 18:48:20.232903 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:20.232888    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:48:30.080702 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:30.080656    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:48:33.294648 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:33.294606    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" event={"ID":"9d058778-1f7c-4d9a-8127-f077dbabff49","Type":"ContainerStarted","Data":"6310dcf2651aeaeedbb672894e02e25a6890ddd7e90ff4e3acd4c24a46ca2b89"}
Aug 04 18:48:33.295089 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:33.294681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" event={"ID":"9d058778-1f7c-4d9a-8127-f077dbabff49","Type":"ContainerStarted","Data":"33ced43c8ebe4af4a3166e612a87c091d9457a129e20e800a9483dc4629b2c2d"}
Aug 04 18:48:33.295089 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:33.294976    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:33.295206 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:33.295115    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:33.296558 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:33.296529    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 04 18:48:33.324641 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:33.324589    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" podStartSLOduration=1.8823763260000002 podStartE2EDuration="18.324572452s" totalImagesPullingTime="12.323628348s" totalInitContainerRuntime="4.118567778s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:48:33.323275573 +0000 UTC m=+2938.667676740" watchObservedRunningTime="2026-08-04 18:48:33.324572452 +0000 UTC m=+2938.668973609"
Aug 04 18:48:34.299730 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:34.299688    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 04 18:48:39.303857 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:39.303828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:48:39.304412 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:39.304384    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 04 18:48:40.080555 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:40.080513    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:48:49.304462 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:49.304422    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 04 18:48:50.081432 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:50.081396    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:48:54.210149 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:54.210098    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:48:59.305136 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:48:59.305097    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 04 18:49:04.210903 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:04.210868    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:49:09.304829 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:09.304780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 04 18:49:09.982336 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:09.982287    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"]
Aug 04 18:49:09.982682 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:09.982644    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kube-rbac-proxy" containerID="cri-o://124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044" gracePeriod=30
Aug 04 18:49:09.982835 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:09.982630    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" containerID="cri-o://316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac" gracePeriod=30
Aug 04 18:49:09.989702 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:09.989675    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:10.075291 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.075236    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 04 18:49:10.148567 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.148536    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"]
Aug 04 18:49:10.151120 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.151104    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.153984 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.153962    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Aug 04 18:49:10.154101 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.154020    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Aug 04 18:49:10.164960 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.164931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"]
Aug 04 18:49:10.219506 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.219464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbc283c5-5e91-425f-9f3e-7d271503920f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.219506 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.219509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2g2p\" (UniqueName: \"kubernetes.io/projected/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-api-access-z2g2p\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.219754 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.219540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.219754 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.219567    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.219754 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.219595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbc283c5-5e91-425f-9f3e-7d271503920f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.234937 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.234863    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:10.320329 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.320293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.320804 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.320340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbc283c5-5e91-425f-9f3e-7d271503920f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.320804 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.320384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbc283c5-5e91-425f-9f3e-7d271503920f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.320804 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.320422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2g2p\" (UniqueName: \"kubernetes.io/projected/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-api-access-z2g2p\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.320804 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.320462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.320804 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.320772    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbc283c5-5e91-425f-9f3e-7d271503920f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.321102 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.321083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.321171 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.321132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.323335 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.323312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbc283c5-5e91-425f-9f3e-7d271503920f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.330120 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.330092    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:10.330594 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.330570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2g2p\" (UniqueName: \"kubernetes.io/projected/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-api-access-z2g2p\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.444672 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.444640    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerID="124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044" exitCode=2
Aug 04 18:49:10.444834 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.444711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" event={"ID":"3a0aae61-4b05-4aa1-b966-8250488dbe1b","Type":"ContainerDied","Data":"124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044"}
Aug 04 18:49:10.460841 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.460810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:10.598896 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:10.598761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"]
Aug 04 18:49:10.601734 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:49:10.601707    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc283c5_5e91_425f_9f3e_7d271503920f.slice/crio-5e37acc2e3bd66356bafd8637217b88cd43905dc412aae6d91a8575abb339495 WatchSource:0}: Error finding container 5e37acc2e3bd66356bafd8637217b88cd43905dc412aae6d91a8575abb339495: Status 404 returned error can't find the container with id 5e37acc2e3bd66356bafd8637217b88cd43905dc412aae6d91a8575abb339495
Aug 04 18:49:11.453280 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:11.453243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" event={"ID":"dbc283c5-5e91-425f-9f3e-7d271503920f","Type":"ContainerStarted","Data":"d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b"}
Aug 04 18:49:11.453763 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:11.453312    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" event={"ID":"dbc283c5-5e91-425f-9f3e-7d271503920f","Type":"ContainerStarted","Data":"5e37acc2e3bd66356bafd8637217b88cd43905dc412aae6d91a8575abb339495"}
Aug 04 18:49:12.458790 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:12.458755    2581 generic.go:356] "Generic (PLEG): container finished" podID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerID="d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b" exitCode=0
Aug 04 18:49:12.459223 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:12.458815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" event={"ID":"dbc283c5-5e91-425f-9f3e-7d271503920f","Type":"ContainerDied","Data":"d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b"}
Aug 04 18:49:13.464782 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:13.464742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" event={"ID":"dbc283c5-5e91-425f-9f3e-7d271503920f","Type":"ContainerStarted","Data":"f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be"}
Aug 04 18:49:13.465229 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:13.464805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" event={"ID":"dbc283c5-5e91-425f-9f3e-7d271503920f","Type":"ContainerStarted","Data":"b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4"}
Aug 04 18:49:13.465229 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:13.464836    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:13.487867 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:13.487805    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podStartSLOduration=2.422363105 podStartE2EDuration="3.487782498s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.065419393s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:49:13.485582788 +0000 UTC m=+2978.829983951" watchObservedRunningTime="2026-08-04 18:49:13.487782498 +0000 UTC m=+2978.832183654"
Aug 04 18:49:13.869021 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:13.868985    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:13.940596 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:13.940555    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:14.210844 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:14.210748    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Aug 04 18:49:14.469603 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:14.469523    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:14.470959 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:14.470930    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 04 18:49:15.075895 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.075853    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Aug 04 18:49:15.245004 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.244977    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:49:15.364811 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.364711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m4h6\" (UniqueName: \"kubernetes.io/projected/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-api-access-2m4h6\") pod \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") "
Aug 04 18:49:15.364811 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.364766    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-rbac-proxy-sar-config\") pod \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") "
Aug 04 18:49:15.364811 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.364811    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kserve-provision-location\") pod \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") "
Aug 04 18:49:15.365152 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.364843    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0aae61-4b05-4aa1-b966-8250488dbe1b-proxy-tls\") pod \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\" (UID: \"3a0aae61-4b05-4aa1-b966-8250488dbe1b\") "
Aug 04 18:49:15.365208 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.365184    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a0aae61-4b05-4aa1-b966-8250488dbe1b" (UID: "3a0aae61-4b05-4aa1-b966-8250488dbe1b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:49:15.365260 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.365188    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a0aae61-4b05-4aa1-b966-8250488dbe1b" (UID: "3a0aae61-4b05-4aa1-b966-8250488dbe1b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:49:15.367105 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.367084    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0aae61-4b05-4aa1-b966-8250488dbe1b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a0aae61-4b05-4aa1-b966-8250488dbe1b" (UID: "3a0aae61-4b05-4aa1-b966-8250488dbe1b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:49:15.367178 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.367148    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-api-access-2m4h6" (OuterVolumeSpecName: "kube-api-access-2m4h6") pod "3a0aae61-4b05-4aa1-b966-8250488dbe1b" (UID: "3a0aae61-4b05-4aa1-b966-8250488dbe1b"). InnerVolumeSpecName "kube-api-access-2m4h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:49:15.465848 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.465809    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:49:15.465848 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.465844    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:49:15.465848 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.465856    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0aae61-4b05-4aa1-b966-8250488dbe1b-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:49:15.466134 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.465865    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m4h6\" (UniqueName: \"kubernetes.io/projected/3a0aae61-4b05-4aa1-b966-8250488dbe1b-kube-api-access-2m4h6\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:49:15.474931 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.474902    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerID="316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac" exitCode=0
Aug 04 18:49:15.475334 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.474972    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"
Aug 04 18:49:15.475334 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.474988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" event={"ID":"3a0aae61-4b05-4aa1-b966-8250488dbe1b","Type":"ContainerDied","Data":"316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac"}
Aug 04 18:49:15.475334 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.475080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6" event={"ID":"3a0aae61-4b05-4aa1-b966-8250488dbe1b","Type":"ContainerDied","Data":"d37d3a0bf2ce3897c3f8b6924df81919c82095ef97e2bd57268bdef835666c0f"}
Aug 04 18:49:15.475334 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.475113    2581 scope.go:117] "RemoveContainer" containerID="124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044"
Aug 04 18:49:15.475705 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.475677    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 04 18:49:15.484512 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.484487    2581 scope.go:117] "RemoveContainer" containerID="316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac"
Aug 04 18:49:15.493775 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.493753    2581 scope.go:117] "RemoveContainer" containerID="eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df"
Aug 04 18:49:15.502477 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.502450    2581 scope.go:117] "RemoveContainer" containerID="124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044"
Aug 04 18:49:15.502842 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:49:15.502819    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044\": container with ID starting with 124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044 not found: ID does not exist" containerID="124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044"
Aug 04 18:49:15.502937 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.502854    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044"} err="failed to get container status \"124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044\": rpc error: code = NotFound desc = could not find container \"124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044\": container with ID starting with 124c3596f80cd16194ed0ce8c92f377332843d137fb79559d1b03cd81ef33044 not found: ID does not exist"
Aug 04 18:49:15.502937 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.502878    2581 scope.go:117] "RemoveContainer" containerID="316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac"
Aug 04 18:49:15.503068 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.502933    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"]
Aug 04 18:49:15.503222 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:49:15.503200    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac\": container with ID starting with 316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac not found: ID does not exist" containerID="316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac"
Aug 04 18:49:15.503303 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.503225    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac"} err="failed to get container status \"316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac\": rpc error: code = NotFound desc = could not find container \"316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac\": container with ID starting with 316f263c029ad8e9832d3ad061be0569a7862c56a9fd5db76fd1bae30dc5cbac not found: ID does not exist"
Aug 04 18:49:15.503303 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.503243    2581 scope.go:117] "RemoveContainer" containerID="eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df"
Aug 04 18:49:15.503487 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:49:15.503472    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df\": container with ID starting with eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df not found: ID does not exist" containerID="eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df"
Aug 04 18:49:15.503532 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.503491    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df"} err="failed to get container status \"eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df\": rpc error: code = NotFound desc = could not find container \"eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df\": container with ID starting with eea79e3a416abc9b7149f271819eafd536a70aa0db6007d2fdd476baad09a4df not found: ID does not exist"
Aug 04 18:49:15.505993 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:15.505972    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbcdbb66-q6cg6"]
Aug 04 18:49:17.214247 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:17.214210    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" path="/var/lib/kubelet/pods/3a0aae61-4b05-4aa1-b966-8250488dbe1b/volumes"
Aug 04 18:49:19.305195 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:19.305164    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:49:20.480750 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:20.480716    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:49:20.481374 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:20.481344    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 04 18:49:26.592613 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:26.592566    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"]
Aug 04 18:49:26.593024 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:26.592915    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kserve-container" containerID="cri-o://33ced43c8ebe4af4a3166e612a87c091d9457a129e20e800a9483dc4629b2c2d" gracePeriod=30
Aug 04 18:49:26.593122 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:26.593004    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kube-rbac-proxy" containerID="cri-o://6310dcf2651aeaeedbb672894e02e25a6890ddd7e90ff4e3acd4c24a46ca2b89" gracePeriod=30
Aug 04 18:49:26.606695 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:26.606622    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:27.527410 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:27.527374    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerID="6310dcf2651aeaeedbb672894e02e25a6890ddd7e90ff4e3acd4c24a46ca2b89" exitCode=2
Aug 04 18:49:27.527614 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:27.527455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" event={"ID":"9d058778-1f7c-4d9a-8127-f077dbabff49","Type":"ContainerDied","Data":"6310dcf2651aeaeedbb672894e02e25a6890ddd7e90ff4e3acd4c24a46ca2b89"}
Aug 04 18:49:29.300792 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.300745    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Aug 04 18:49:29.305206 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.305174    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Aug 04 18:49:29.539777 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.539736    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerID="33ced43c8ebe4af4a3166e612a87c091d9457a129e20e800a9483dc4629b2c2d" exitCode=0
Aug 04 18:49:29.539976 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.539853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" event={"ID":"9d058778-1f7c-4d9a-8127-f077dbabff49","Type":"ContainerDied","Data":"33ced43c8ebe4af4a3166e612a87c091d9457a129e20e800a9483dc4629b2c2d"}
Aug 04 18:49:29.647539 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.647508    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:49:29.790176 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.790137    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spq66\" (UniqueName: \"kubernetes.io/projected/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-api-access-spq66\") pod \"9d058778-1f7c-4d9a-8127-f077dbabff49\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") "
Aug 04 18:49:29.790390 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.790201    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d058778-1f7c-4d9a-8127-f077dbabff49-proxy-tls\") pod \"9d058778-1f7c-4d9a-8127-f077dbabff49\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") "
Aug 04 18:49:29.790390 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.790231    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-rbac-proxy-sar-config\") pod \"9d058778-1f7c-4d9a-8127-f077dbabff49\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") "
Aug 04 18:49:29.790390 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.790271    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d058778-1f7c-4d9a-8127-f077dbabff49-kserve-provision-location\") pod \"9d058778-1f7c-4d9a-8127-f077dbabff49\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") "
Aug 04 18:49:29.790573 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.790411    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-cabundle-cert\") pod \"9d058778-1f7c-4d9a-8127-f077dbabff49\" (UID: \"9d058778-1f7c-4d9a-8127-f077dbabff49\") "
Aug 04 18:49:29.790677 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.790637    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d058778-1f7c-4d9a-8127-f077dbabff49" (UID: "9d058778-1f7c-4d9a-8127-f077dbabff49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:49:29.790746 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.790712    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d058778-1f7c-4d9a-8127-f077dbabff49" (UID: "9d058778-1f7c-4d9a-8127-f077dbabff49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:49:29.792541 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.792514    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d058778-1f7c-4d9a-8127-f077dbabff49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d058778-1f7c-4d9a-8127-f077dbabff49" (UID: "9d058778-1f7c-4d9a-8127-f077dbabff49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:49:29.792660 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.792599    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-api-access-spq66" (OuterVolumeSpecName: "kube-api-access-spq66") pod "9d058778-1f7c-4d9a-8127-f077dbabff49" (UID: "9d058778-1f7c-4d9a-8127-f077dbabff49"). InnerVolumeSpecName "kube-api-access-spq66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:49:29.803692 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.803648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d058778-1f7c-4d9a-8127-f077dbabff49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d058778-1f7c-4d9a-8127-f077dbabff49" (UID: "9d058778-1f7c-4d9a-8127-f077dbabff49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:49:29.891608 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.891492    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:49:29.891608 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.891524    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spq66\" (UniqueName: \"kubernetes.io/projected/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-api-access-spq66\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:49:29.891608 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.891533    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d058778-1f7c-4d9a-8127-f077dbabff49-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:49:29.891608 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.891542    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d058778-1f7c-4d9a-8127-f077dbabff49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:49:29.891608 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:29.891552    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d058778-1f7c-4d9a-8127-f077dbabff49-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:49:30.481659 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:30.481623    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 04 18:49:30.545957 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:30.545925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp" event={"ID":"9d058778-1f7c-4d9a-8127-f077dbabff49","Type":"ContainerDied","Data":"2d3dfa16eda1ae0a0e518f6ba15cac1e48eaae06082a3a91cccd09350addaf3c"}
Aug 04 18:49:30.545957 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:30.545945    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"
Aug 04 18:49:30.546195 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:30.545991    2581 scope.go:117] "RemoveContainer" containerID="6310dcf2651aeaeedbb672894e02e25a6890ddd7e90ff4e3acd4c24a46ca2b89"
Aug 04 18:49:30.554675 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:30.554511    2581 scope.go:117] "RemoveContainer" containerID="33ced43c8ebe4af4a3166e612a87c091d9457a129e20e800a9483dc4629b2c2d"
Aug 04 18:49:30.562561 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:30.562540    2581 scope.go:117] "RemoveContainer" containerID="e431dff4372c6e0c2164687be25ca1b5f338134c56dc538de3e4bd29c02fe027"
Aug 04 18:49:30.570648 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:30.570621    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"]
Aug 04 18:49:30.578833 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:30.578803    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vztxp"]
Aug 04 18:49:31.213952 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:31.213915    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" path="/var/lib/kubelet/pods/9d058778-1f7c-4d9a-8127-f077dbabff49/volumes"
Aug 04 18:49:31.732283 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:31.732242    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:31.838442 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:31.838404    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:49:35.357663 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:35.357634    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:49:35.358286 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:35.358267    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:49:35.364769 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:35.364746    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:49:35.365359 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:35.365338    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:49:40.481237 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:40.481198    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 04 18:49:50.482068 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:49:50.481999    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 04 18:50:00.481709 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:00.481669    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 04 18:50:10.481278 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:10.481235    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 04 18:50:11.040305 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:11.040271    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:20.482240 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:20.482193    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 04 18:50:28.209790 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:28.209701    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Aug 04 18:50:38.211177 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:38.211145    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:50:40.182019 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:40.181986    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"]
Aug 04 18:50:40.182506 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:40.182292    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kube-rbac-proxy" containerID="cri-o://f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be" gracePeriod=30
Aug 04 18:50:40.182506 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:40.182292    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container" containerID="cri-o://b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4" gracePeriod=30
Aug 04 18:50:40.216866 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:40.216818    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:40.476412 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:40.476321    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Aug 04 18:50:40.841810 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:40.841771    2581 generic.go:356] "Generic (PLEG): container finished" podID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerID="f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be" exitCode=2
Aug 04 18:50:40.842008 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:40.841842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" event={"ID":"dbc283c5-5e91-425f-9f3e-7d271503920f","Type":"ContainerDied","Data":"f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be"}
Aug 04 18:50:41.267698 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.267590    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:41.300269 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300228    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"]
Aug 04 18:50:41.300557 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300542    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kserve-container"
Aug 04 18:50:41.300597 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300559    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kserve-container"
Aug 04 18:50:41.300597 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300572    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="storage-initializer"
Aug 04 18:50:41.300597 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300582    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="storage-initializer"
Aug 04 18:50:41.300597 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300594    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kube-rbac-proxy"
Aug 04 18:50:41.300728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300600    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kube-rbac-proxy"
Aug 04 18:50:41.300728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300616    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kube-rbac-proxy"
Aug 04 18:50:41.300728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300621    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kube-rbac-proxy"
Aug 04 18:50:41.300728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300627    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="storage-initializer"
Aug 04 18:50:41.300728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300631    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="storage-initializer"
Aug 04 18:50:41.300728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300641    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container"
Aug 04 18:50:41.300728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300646    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container"
Aug 04 18:50:41.300728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300701    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kserve-container"
Aug 04 18:50:41.300728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300708    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0aae61-4b05-4aa1-b966-8250488dbe1b" containerName="kube-rbac-proxy"
Aug 04 18:50:41.300728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300716    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kube-rbac-proxy"
Aug 04 18:50:41.300728 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.300725    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d058778-1f7c-4d9a-8127-f077dbabff49" containerName="kserve-container"
Aug 04 18:50:41.303248 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.303226    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.305491 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.305464    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Aug 04 18:50:41.305621 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.305471    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Aug 04 18:50:41.314165 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.314135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"]
Aug 04 18:50:41.483411 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.483374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1de8076-c276-4b42-b043-2af9e04f4d7b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.483589 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.483417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.483589 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.483530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcttx\" (UniqueName: \"kubernetes.io/projected/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-api-access-pcttx\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.483664 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.483598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1de8076-c276-4b42-b043-2af9e04f4d7b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.585071 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.584993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1de8076-c276-4b42-b043-2af9e04f4d7b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.585261 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.585088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.585261 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.585137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcttx\" (UniqueName: \"kubernetes.io/projected/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-api-access-pcttx\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.585261 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.585168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1de8076-c276-4b42-b043-2af9e04f4d7b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.585261 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:50:41.585175    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Aug 04 18:50:41.585467 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:50:41.585263    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1de8076-c276-4b42-b043-2af9e04f4d7b-proxy-tls podName:c1de8076-c276-4b42-b043-2af9e04f4d7b nodeName:}" failed. No retries permitted until 2026-08-04 18:50:42.085242861 +0000 UTC m=+3067.429644016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c1de8076-c276-4b42-b043-2af9e04f4d7b-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh" (UID: "c1de8076-c276-4b42-b043-2af9e04f4d7b") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Aug 04 18:50:41.585546 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.585529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1de8076-c276-4b42-b043-2af9e04f4d7b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.585759 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.585741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.594718 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.594696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcttx\" (UniqueName: \"kubernetes.io/projected/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-api-access-pcttx\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:41.802501 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:41.802467    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:42.089382 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:42.089344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1de8076-c276-4b42-b043-2af9e04f4d7b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:42.092215 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:42.092192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1de8076-c276-4b42-b043-2af9e04f4d7b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:42.215951 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:42.215909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:42.358608 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:42.358579    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"]
Aug 04 18:50:42.361388 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:50:42.361343    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1de8076_c276_4b42_b043_2af9e04f4d7b.slice/crio-d23add246f372ded81aa7ecb7de7a88f6912ed41bf4d56877ccd30406e68b880 WatchSource:0}: Error finding container d23add246f372ded81aa7ecb7de7a88f6912ed41bf4d56877ccd30406e68b880: Status 404 returned error can't find the container with id d23add246f372ded81aa7ecb7de7a88f6912ed41bf4d56877ccd30406e68b880
Aug 04 18:50:42.852019 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:42.851976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh" event={"ID":"c1de8076-c276-4b42-b043-2af9e04f4d7b","Type":"ContainerStarted","Data":"ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0"}
Aug 04 18:50:42.852224 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:42.852082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh" event={"ID":"c1de8076-c276-4b42-b043-2af9e04f4d7b","Type":"ContainerStarted","Data":"d23add246f372ded81aa7ecb7de7a88f6912ed41bf4d56877ccd30406e68b880"}
Aug 04 18:50:43.354398 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:43.354355    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:45.137549 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.137518    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:50:45.209914 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.209879    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbc283c5-5e91-425f-9f3e-7d271503920f-kserve-provision-location\") pod \"dbc283c5-5e91-425f-9f3e-7d271503920f\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") "
Aug 04 18:50:45.210137 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.209937    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-rbac-proxy-sar-config\") pod \"dbc283c5-5e91-425f-9f3e-7d271503920f\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") "
Aug 04 18:50:45.210137 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.209958    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-cabundle-cert\") pod \"dbc283c5-5e91-425f-9f3e-7d271503920f\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") "
Aug 04 18:50:45.210137 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.209997    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbc283c5-5e91-425f-9f3e-7d271503920f-proxy-tls\") pod \"dbc283c5-5e91-425f-9f3e-7d271503920f\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") "
Aug 04 18:50:45.210137 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.210025    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2g2p\" (UniqueName: \"kubernetes.io/projected/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-api-access-z2g2p\") pod \"dbc283c5-5e91-425f-9f3e-7d271503920f\" (UID: \"dbc283c5-5e91-425f-9f3e-7d271503920f\") "
Aug 04 18:50:45.210442 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.210285    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc283c5-5e91-425f-9f3e-7d271503920f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbc283c5-5e91-425f-9f3e-7d271503920f" (UID: "dbc283c5-5e91-425f-9f3e-7d271503920f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:50:45.210442 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.210297    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dbc283c5-5e91-425f-9f3e-7d271503920f" (UID: "dbc283c5-5e91-425f-9f3e-7d271503920f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:50:45.210442 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.210416    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dbc283c5-5e91-425f-9f3e-7d271503920f" (UID: "dbc283c5-5e91-425f-9f3e-7d271503920f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:50:45.212512 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.212486    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc283c5-5e91-425f-9f3e-7d271503920f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbc283c5-5e91-425f-9f3e-7d271503920f" (UID: "dbc283c5-5e91-425f-9f3e-7d271503920f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:50:45.212634 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.212553    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-api-access-z2g2p" (OuterVolumeSpecName: "kube-api-access-z2g2p") pod "dbc283c5-5e91-425f-9f3e-7d271503920f" (UID: "dbc283c5-5e91-425f-9f3e-7d271503920f"). InnerVolumeSpecName "kube-api-access-z2g2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:50:45.310629 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.310591    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbc283c5-5e91-425f-9f3e-7d271503920f-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:50:45.310629 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.310626    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:50:45.310849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.310641    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbc283c5-5e91-425f-9f3e-7d271503920f-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:50:45.310849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.310668    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbc283c5-5e91-425f-9f3e-7d271503920f-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:50:45.310849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.310678    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2g2p\" (UniqueName: \"kubernetes.io/projected/dbc283c5-5e91-425f-9f3e-7d271503920f-kube-api-access-z2g2p\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:50:45.865780 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.865740    2581 generic.go:356] "Generic (PLEG): container finished" podID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerID="b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4" exitCode=0
Aug 04 18:50:45.865962 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.865796    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"
Aug 04 18:50:45.865962 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.865820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" event={"ID":"dbc283c5-5e91-425f-9f3e-7d271503920f","Type":"ContainerDied","Data":"b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4"}
Aug 04 18:50:45.865962 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.865872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh" event={"ID":"dbc283c5-5e91-425f-9f3e-7d271503920f","Type":"ContainerDied","Data":"5e37acc2e3bd66356bafd8637217b88cd43905dc412aae6d91a8575abb339495"}
Aug 04 18:50:45.865962 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.865897    2581 scope.go:117] "RemoveContainer" containerID="f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be"
Aug 04 18:50:45.874641 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.874620    2581 scope.go:117] "RemoveContainer" containerID="b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4"
Aug 04 18:50:45.882994 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.882973    2581 scope.go:117] "RemoveContainer" containerID="d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b"
Aug 04 18:50:45.887661 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.887632    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"]
Aug 04 18:50:45.891998 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.891904    2581 scope.go:117] "RemoveContainer" containerID="f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be"
Aug 04 18:50:45.892448 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:50:45.892423    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be\": container with ID starting with f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be not found: ID does not exist" containerID="f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be"
Aug 04 18:50:45.892527 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.892462    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be"} err="failed to get container status \"f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be\": rpc error: code = NotFound desc = could not find container \"f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be\": container with ID starting with f92f09edf37f950024116e329d5ba9d5ecc870ebd88113d5d7a6d12a38aa19be not found: ID does not exist"
Aug 04 18:50:45.892527 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.892483    2581 scope.go:117] "RemoveContainer" containerID="b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4"
Aug 04 18:50:45.892780 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:50:45.892762    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4\": container with ID starting with b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4 not found: ID does not exist" containerID="b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4"
Aug 04 18:50:45.892834 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.892784    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4"} err="failed to get container status \"b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4\": rpc error: code = NotFound desc = could not find container \"b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4\": container with ID starting with b2d84e165f11f921d8a9ad45e7df5c4eb7875022147ffdf3ff0f9db8094715f4 not found: ID does not exist"
Aug 04 18:50:45.892834 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.892798    2581 scope.go:117] "RemoveContainer" containerID="d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b"
Aug 04 18:50:45.893062 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:50:45.893028    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b\": container with ID starting with d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b not found: ID does not exist" containerID="d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b"
Aug 04 18:50:45.893114 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.893067    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b"} err="failed to get container status \"d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b\": rpc error: code = NotFound desc = could not find container \"d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b\": container with ID starting with d90574ef6bbc99f5df15509af03b61d25fe5523d57c90e579091903d5da2ef2b not found: ID does not exist"
Aug 04 18:50:45.893482 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:45.893464    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-86b6cc7886-x7zdh"]
Aug 04 18:50:47.214655 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:47.214620    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" path="/var/lib/kubelet/pods/dbc283c5-5e91-425f-9f3e-7d271503920f/volumes"
Aug 04 18:50:47.878983 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:47.878885    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh_c1de8076-c276-4b42-b043-2af9e04f4d7b/storage-initializer/0.log"
Aug 04 18:50:47.878983 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:47.878930    2581 generic.go:356] "Generic (PLEG): container finished" podID="c1de8076-c276-4b42-b043-2af9e04f4d7b" containerID="ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0" exitCode=1
Aug 04 18:50:47.879250 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:47.879067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh" event={"ID":"c1de8076-c276-4b42-b043-2af9e04f4d7b","Type":"ContainerDied","Data":"ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0"}
Aug 04 18:50:48.460938 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:48.460905    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:48.884745 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:48.884716    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh_c1de8076-c276-4b42-b043-2af9e04f4d7b/storage-initializer/0.log"
Aug 04 18:50:48.884920 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:48.884784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh" event={"ID":"c1de8076-c276-4b42-b043-2af9e04f4d7b","Type":"ContainerStarted","Data":"d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16"}
Aug 04 18:50:48.934125 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:48.934081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:51.336261 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:51.336227    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"]
Aug 04 18:50:51.336671 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:51.336451    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh" podUID="c1de8076-c276-4b42-b043-2af9e04f4d7b" containerName="storage-initializer" containerID="cri-o://d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16" gracePeriod=30
Aug 04 18:50:51.360393 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:51.360353    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:52.429604 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.429558    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:52.437185 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.437154    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"]
Aug 04 18:50:52.437465 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.437453    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="storage-initializer"
Aug 04 18:50:52.437508 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.437466    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="storage-initializer"
Aug 04 18:50:52.437508 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.437479    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container"
Aug 04 18:50:52.437508 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.437485    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container"
Aug 04 18:50:52.437508 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.437492    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kube-rbac-proxy"
Aug 04 18:50:52.437508 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.437498    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kube-rbac-proxy"
Aug 04 18:50:52.437663 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.437552    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kube-rbac-proxy"
Aug 04 18:50:52.437663 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.437559    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc283c5-5e91-425f-9f3e-7d271503920f" containerName="kserve-container"
Aug 04 18:50:52.440799 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.440774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.443299 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.443275    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:50:52.443474 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.443319    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Aug 04 18:50:52.443555 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.443542    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Aug 04 18:50:52.452314 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.452286    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"]
Aug 04 18:50:52.459494 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.459452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.459685 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.459523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7tnv\" (UniqueName: \"kubernetes.io/projected/31ca71df-78d0-4f87-8801-18b32239d44c-kube-api-access-g7tnv\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.459685 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.459549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.459685 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.459579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31ca71df-78d0-4f87-8801-18b32239d44c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.459685 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.459640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ca71df-78d0-4f87-8801-18b32239d44c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.560410 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.560368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7tnv\" (UniqueName: \"kubernetes.io/projected/31ca71df-78d0-4f87-8801-18b32239d44c-kube-api-access-g7tnv\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.560410 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.560408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.560691 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.560433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31ca71df-78d0-4f87-8801-18b32239d44c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.560691 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.560465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ca71df-78d0-4f87-8801-18b32239d44c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.560691 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.560544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.561162 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.561134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.561325 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.561165    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ca71df-78d0-4f87-8801-18b32239d44c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.561325 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.561231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.563228 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.563205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31ca71df-78d0-4f87-8801-18b32239d44c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.570629 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.570596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:52.571958 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.571935    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7tnv\" (UniqueName: \"kubernetes.io/projected/31ca71df-78d0-4f87-8801-18b32239d44c-kube-api-access-g7tnv\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.753011 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.752969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:52.789208 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.789181    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh_c1de8076-c276-4b42-b043-2af9e04f4d7b/storage-initializer/1.log"
Aug 04 18:50:52.789655 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.789640    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh_c1de8076-c276-4b42-b043-2af9e04f4d7b/storage-initializer/0.log"
Aug 04 18:50:52.791605 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.791581    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:52.863354 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.863322    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1de8076-c276-4b42-b043-2af9e04f4d7b-kserve-provision-location\") pod \"c1de8076-c276-4b42-b043-2af9e04f4d7b\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") "
Aug 04 18:50:52.863545 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.863375    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcttx\" (UniqueName: \"kubernetes.io/projected/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-api-access-pcttx\") pod \"c1de8076-c276-4b42-b043-2af9e04f4d7b\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") "
Aug 04 18:50:52.863545 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.863517    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1de8076-c276-4b42-b043-2af9e04f4d7b-proxy-tls\") pod \"c1de8076-c276-4b42-b043-2af9e04f4d7b\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") "
Aug 04 18:50:52.863666 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.863551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-rbac-proxy-sar-config\") pod \"c1de8076-c276-4b42-b043-2af9e04f4d7b\" (UID: \"c1de8076-c276-4b42-b043-2af9e04f4d7b\") "
Aug 04 18:50:52.863666 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.863644    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1de8076-c276-4b42-b043-2af9e04f4d7b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1de8076-c276-4b42-b043-2af9e04f4d7b" (UID: "c1de8076-c276-4b42-b043-2af9e04f4d7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:50:52.863799 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.863780    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1de8076-c276-4b42-b043-2af9e04f4d7b-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:50:52.864109 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.864084    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c1de8076-c276-4b42-b043-2af9e04f4d7b" (UID: "c1de8076-c276-4b42-b043-2af9e04f4d7b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:50:52.866084 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.866000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-api-access-pcttx" (OuterVolumeSpecName: "kube-api-access-pcttx") pod "c1de8076-c276-4b42-b043-2af9e04f4d7b" (UID: "c1de8076-c276-4b42-b043-2af9e04f4d7b"). InnerVolumeSpecName "kube-api-access-pcttx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:50:52.866407 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.866377    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1de8076-c276-4b42-b043-2af9e04f4d7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1de8076-c276-4b42-b043-2af9e04f4d7b" (UID: "c1de8076-c276-4b42-b043-2af9e04f4d7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:50:52.901197 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.901170    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh_c1de8076-c276-4b42-b043-2af9e04f4d7b/storage-initializer/1.log"
Aug 04 18:50:52.901557 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.901539    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh_c1de8076-c276-4b42-b043-2af9e04f4d7b/storage-initializer/0.log"
Aug 04 18:50:52.901641 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.901578    2581 generic.go:356] "Generic (PLEG): container finished" podID="c1de8076-c276-4b42-b043-2af9e04f4d7b" containerID="d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16" exitCode=1
Aug 04 18:50:52.901641 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.901613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh" event={"ID":"c1de8076-c276-4b42-b043-2af9e04f4d7b","Type":"ContainerDied","Data":"d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16"}
Aug 04 18:50:52.901743 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.901663    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"
Aug 04 18:50:52.901743 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.901674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh" event={"ID":"c1de8076-c276-4b42-b043-2af9e04f4d7b","Type":"ContainerDied","Data":"d23add246f372ded81aa7ecb7de7a88f6912ed41bf4d56877ccd30406e68b880"}
Aug 04 18:50:52.901743 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.901702    2581 scope.go:117] "RemoveContainer" containerID="d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16"
Aug 04 18:50:52.908427 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.908312    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"]
Aug 04 18:50:52.911904 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.911878    2581 scope.go:117] "RemoveContainer" containerID="ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0"
Aug 04 18:50:52.913152 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:50:52.913127    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ca71df_78d0_4f87_8801_18b32239d44c.slice/crio-acda1b7a9ac45bdab3d27d34cc6c2186ad1ccabaf5810744db56c25819cde24e WatchSource:0}: Error finding container acda1b7a9ac45bdab3d27d34cc6c2186ad1ccabaf5810744db56c25819cde24e: Status 404 returned error can't find the container with id acda1b7a9ac45bdab3d27d34cc6c2186ad1ccabaf5810744db56c25819cde24e
Aug 04 18:50:52.920358 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.920335    2581 scope.go:117] "RemoveContainer" containerID="d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16"
Aug 04 18:50:52.920683 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:50:52.920660    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16\": container with ID starting with d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16 not found: ID does not exist" containerID="d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16"
Aug 04 18:50:52.920782 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.920689    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16"} err="failed to get container status \"d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16\": rpc error: code = NotFound desc = could not find container \"d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16\": container with ID starting with d3d5c93448bdeb46e84f53f8453c8d89e2f00ee7a56e03a2cf8efdaefc686e16 not found: ID does not exist"
Aug 04 18:50:52.920782 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.920708    2581 scope.go:117] "RemoveContainer" containerID="ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0"
Aug 04 18:50:52.920963 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:50:52.920946    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0\": container with ID starting with ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0 not found: ID does not exist" containerID="ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0"
Aug 04 18:50:52.921007 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.920967    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0"} err="failed to get container status \"ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0\": rpc error: code = NotFound desc = could not find container \"ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0\": container with ID starting with ef10eeb39b940e8a4eb574cc2d8cbf1139cfa471352f78b51d24c0207ff07bb0 not found: ID does not exist"
Aug 04 18:50:52.943967 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.943939    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"]
Aug 04 18:50:52.949214 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.949185    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5976fc8486-jhhqh"]
Aug 04 18:50:52.964138 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.964113    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1de8076-c276-4b42-b043-2af9e04f4d7b-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:50:52.964138 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.964137    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:50:52.964290 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:52.964147    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcttx\" (UniqueName: \"kubernetes.io/projected/c1de8076-c276-4b42-b043-2af9e04f4d7b-kube-api-access-pcttx\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:50:53.221010 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:53.220924    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1de8076-c276-4b42-b043-2af9e04f4d7b" path="/var/lib/kubelet/pods/c1de8076-c276-4b42-b043-2af9e04f4d7b/volumes"
Aug 04 18:50:53.908384 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:53.908289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" event={"ID":"31ca71df-78d0-4f87-8801-18b32239d44c","Type":"ContainerStarted","Data":"467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb"}
Aug 04 18:50:53.908384 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:53.908357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" event={"ID":"31ca71df-78d0-4f87-8801-18b32239d44c","Type":"ContainerStarted","Data":"acda1b7a9ac45bdab3d27d34cc6c2186ad1ccabaf5810744db56c25819cde24e"}
Aug 04 18:50:54.914098 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:54.914037    2581 generic.go:356] "Generic (PLEG): container finished" podID="31ca71df-78d0-4f87-8801-18b32239d44c" containerID="467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb" exitCode=0
Aug 04 18:50:54.914520 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:54.914119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" event={"ID":"31ca71df-78d0-4f87-8801-18b32239d44c","Type":"ContainerDied","Data":"467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb"}
Aug 04 18:50:55.920690 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:55.920653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" event={"ID":"31ca71df-78d0-4f87-8801-18b32239d44c","Type":"ContainerStarted","Data":"bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d"}
Aug 04 18:50:55.921072 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:55.920713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" event={"ID":"31ca71df-78d0-4f87-8801-18b32239d44c","Type":"ContainerStarted","Data":"d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca"}
Aug 04 18:50:55.921072 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:55.920799    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:55.921072 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:55.920860    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:50:55.922375 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:55.922342    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 04 18:50:55.942465 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:55.942411    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podStartSLOduration=2.887382161 podStartE2EDuration="3.942386257s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.055004096s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:50:55.942012953 +0000 UTC m=+3081.286414111" watchObservedRunningTime="2026-08-04 18:50:55.942386257 +0000 UTC m=+3081.286787413"
Aug 04 18:50:56.925197 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:50:56.925148    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 04 18:51:01.929713 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:01.929684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:51:01.930241 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:01.930216    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 04 18:51:11.931130 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:11.931079    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 04 18:51:21.930258 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:21.930211    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 04 18:51:31.930326 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:31.930280    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 04 18:51:41.930405 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:41.930365    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 04 18:51:50.461161 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:50.461120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:51.930319 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:51.930272    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 04 18:51:55.571267 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.571229    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:55.612825 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.612790    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"]
Aug 04 18:51:55.613127 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.613110    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1de8076-c276-4b42-b043-2af9e04f4d7b" containerName="storage-initializer"
Aug 04 18:51:55.613127 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.613127    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1de8076-c276-4b42-b043-2af9e04f4d7b" containerName="storage-initializer"
Aug 04 18:51:55.613264 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.613143    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1de8076-c276-4b42-b043-2af9e04f4d7b" containerName="storage-initializer"
Aug 04 18:51:55.613264 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.613149    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1de8076-c276-4b42-b043-2af9e04f4d7b" containerName="storage-initializer"
Aug 04 18:51:55.613264 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.613229    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1de8076-c276-4b42-b043-2af9e04f4d7b" containerName="storage-initializer"
Aug 04 18:51:55.613369 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.613334    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1de8076-c276-4b42-b043-2af9e04f4d7b" containerName="storage-initializer"
Aug 04 18:51:55.616286 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.616266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.619645 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.619618    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:51:55.619776 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.619619    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Aug 04 18:51:55.619776 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.619619    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Aug 04 18:51:55.628449 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.628421    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"]
Aug 04 18:51:55.722218 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.722180    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:55.783228 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.783151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d195bcb-e83d-44f5-a550-03c0f4086de4-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.783228 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.783209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.783419 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.783264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d195bcb-e83d-44f5-a550-03c0f4086de4-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.783419 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.783332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.783419 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.783383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvh95\" (UniqueName: \"kubernetes.io/projected/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-api-access-gvh95\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.883867 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.883776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d195bcb-e83d-44f5-a550-03c0f4086de4-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.883867 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.883834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.883867 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.883871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d195bcb-e83d-44f5-a550-03c0f4086de4-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.884176 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.883893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.884176 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.883922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvh95\" (UniqueName: \"kubernetes.io/projected/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-api-access-gvh95\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.884375 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.884348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d195bcb-e83d-44f5-a550-03c0f4086de4-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.884644 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.884627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.884737 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.884708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.886748 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.886725    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d195bcb-e83d-44f5-a550-03c0f4086de4-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.894626 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.894599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvh95\" (UniqueName: \"kubernetes.io/projected/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-api-access-gvh95\") pod \"isvc-pmml-predictor-8674795f59-c7rx4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:55.927558 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:55.927518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:51:56.078994 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:56.078966    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"]
Aug 04 18:51:56.081668 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:51:56.081623    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d195bcb_e83d_44f5_a550_03c0f4086de4.slice/crio-0530ad92f183d07ff9e935f5148ee1603b9998f26f755eeaf3a8657045cc9bc4 WatchSource:0}: Error finding container 0530ad92f183d07ff9e935f5148ee1603b9998f26f755eeaf3a8657045cc9bc4: Status 404 returned error can't find the container with id 0530ad92f183d07ff9e935f5148ee1603b9998f26f755eeaf3a8657045cc9bc4
Aug 04 18:51:56.160831 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:56.160797    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" event={"ID":"1d195bcb-e83d-44f5-a550-03c0f4086de4","Type":"ContainerStarted","Data":"0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0"}
Aug 04 18:51:56.161075 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:51:56.160859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" event={"ID":"1d195bcb-e83d-44f5-a550-03c0f4086de4","Type":"ContainerStarted","Data":"0530ad92f183d07ff9e935f5148ee1603b9998f26f755eeaf3a8657045cc9bc4"}
Aug 04 18:52:00.177457 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:00.177417    2581 generic.go:356] "Generic (PLEG): container finished" podID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerID="0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0" exitCode=0
Aug 04 18:52:00.177959 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:00.177504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" event={"ID":"1d195bcb-e83d-44f5-a550-03c0f4086de4","Type":"ContainerDied","Data":"0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0"}
Aug 04 18:52:01.931454 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:01.931408    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Aug 04 18:52:08.219733 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:08.219691    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" event={"ID":"1d195bcb-e83d-44f5-a550-03c0f4086de4","Type":"ContainerStarted","Data":"475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614"}
Aug 04 18:52:08.220144 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:08.219764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" event={"ID":"1d195bcb-e83d-44f5-a550-03c0f4086de4","Type":"ContainerStarted","Data":"2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830"}
Aug 04 18:52:08.220144 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:08.219972    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:52:08.243155 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:08.243103    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podStartSLOduration=2.441925796 podStartE2EDuration="13.243084688s" totalImagesPullingTime="7.251582305s" totalInitContainerRuntime="3.549576587s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:52:08.242135429 +0000 UTC m=+3153.586536588" watchObservedRunningTime="2026-08-04 18:52:08.243084688 +0000 UTC m=+3153.587485847"
Aug 04 18:52:09.223566 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:09.223530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:52:09.224985 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:09.224954    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 04 18:52:10.228030 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:10.227990    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 04 18:52:11.930852 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:11.930825    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:52:12.476153 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:12.476118    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"]
Aug 04 18:52:12.476466 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:12.476431    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container" containerID="cri-o://d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca" gracePeriod=30
Aug 04 18:52:12.476608 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:12.476496    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kube-rbac-proxy" containerID="cri-o://bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d" gracePeriod=30
Aug 04 18:52:12.506789 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:12.506760    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:52:13.241886 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.241851    2581 generic.go:356] "Generic (PLEG): container finished" podID="31ca71df-78d0-4f87-8801-18b32239d44c" containerID="bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d" exitCode=2
Aug 04 18:52:13.242328 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.241912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" event={"ID":"31ca71df-78d0-4f87-8801-18b32239d44c","Type":"ContainerDied","Data":"bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d"}
Aug 04 18:52:13.572857 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.572818    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"]
Aug 04 18:52:13.574435 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.574410    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:52:13.576348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.576334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:13.579073 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.579022    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Aug 04 18:52:13.581984 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.581954    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Aug 04 18:52:13.591102 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.591033    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"]
Aug 04 18:52:13.632669 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.632632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2w4\" (UniqueName: \"kubernetes.io/projected/087dc74d-637b-4313-86dc-4082f4a72958-kube-api-access-pn2w4\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:13.632841 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.632686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/087dc74d-637b-4313-86dc-4082f4a72958-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:13.632841 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.632754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/087dc74d-637b-4313-86dc-4082f4a72958-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:13.632841 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.632798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/087dc74d-637b-4313-86dc-4082f4a72958-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:13.733503 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.733467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/087dc74d-637b-4313-86dc-4082f4a72958-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:13.733711 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.733519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/087dc74d-637b-4313-86dc-4082f4a72958-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:13.733711 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.733548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/087dc74d-637b-4313-86dc-4082f4a72958-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:13.733711 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.733583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2w4\" (UniqueName: \"kubernetes.io/projected/087dc74d-637b-4313-86dc-4082f4a72958-kube-api-access-pn2w4\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:13.733896 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:52:13.733719    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Aug 04 18:52:13.733896 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:52:13.733810    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/087dc74d-637b-4313-86dc-4082f4a72958-proxy-tls podName:087dc74d-637b-4313-86dc-4082f4a72958 nodeName:}" failed. No retries permitted until 2026-08-04 18:52:14.233787583 +0000 UTC m=+3159.578188728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/087dc74d-637b-4313-86dc-4082f4a72958-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422" (UID: "087dc74d-637b-4313-86dc-4082f4a72958") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Aug 04 18:52:13.733992 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.733926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/087dc74d-637b-4313-86dc-4082f4a72958-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:13.734350 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.734330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/087dc74d-637b-4313-86dc-4082f4a72958-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:13.743810 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:13.743777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2w4\" (UniqueName: \"kubernetes.io/projected/087dc74d-637b-4313-86dc-4082f4a72958-kube-api-access-pn2w4\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:14.167804 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:14.167770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:52:14.236412 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:14.236368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/087dc74d-637b-4313-86dc-4082f4a72958-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:14.239076 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:14.239014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/087dc74d-637b-4313-86dc-4082f4a72958-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:14.490824 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:14.490713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:14.635059 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:14.635014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"]
Aug 04 18:52:14.637621 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:52:14.637592    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087dc74d_637b_4313_86dc_4082f4a72958.slice/crio-1880bde0b777f3c33afdf07b842c224c909e2512658165d7f2d4ce6230c01a3a WatchSource:0}: Error finding container 1880bde0b777f3c33afdf07b842c224c909e2512658165d7f2d4ce6230c01a3a: Status 404 returned error can't find the container with id 1880bde0b777f3c33afdf07b842c224c909e2512658165d7f2d4ce6230c01a3a
Aug 04 18:52:15.234718 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:15.234684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:52:15.235342 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:15.235311    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 04 18:52:15.252617 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:15.252583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422" event={"ID":"087dc74d-637b-4313-86dc-4082f4a72958","Type":"ContainerStarted","Data":"125abab5c9bca348d76eb73ce5192f40c25b1c76201d4312bc867a0570438a6e"}
Aug 04 18:52:15.252839 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:15.252684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422" event={"ID":"087dc74d-637b-4313-86dc-4082f4a72958","Type":"ContainerStarted","Data":"1880bde0b777f3c33afdf07b842c224c909e2512658165d7f2d4ce6230c01a3a"}
Aug 04 18:52:16.926234 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:16.926194    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Aug 04 18:52:17.840391 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.840357    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:52:17.859827 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.859751    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31ca71df-78d0-4f87-8801-18b32239d44c-proxy-tls\") pod \"31ca71df-78d0-4f87-8801-18b32239d44c\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") "
Aug 04 18:52:17.859827 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.859790    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-cabundle-cert\") pod \"31ca71df-78d0-4f87-8801-18b32239d44c\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") "
Aug 04 18:52:17.859827 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.859814    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ca71df-78d0-4f87-8801-18b32239d44c-kserve-provision-location\") pod \"31ca71df-78d0-4f87-8801-18b32239d44c\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") "
Aug 04 18:52:17.859987 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.859848    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7tnv\" (UniqueName: \"kubernetes.io/projected/31ca71df-78d0-4f87-8801-18b32239d44c-kube-api-access-g7tnv\") pod \"31ca71df-78d0-4f87-8801-18b32239d44c\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") "
Aug 04 18:52:17.859987 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.859892    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-kube-rbac-proxy-sar-config\") pod \"31ca71df-78d0-4f87-8801-18b32239d44c\" (UID: \"31ca71df-78d0-4f87-8801-18b32239d44c\") "
Aug 04 18:52:17.860251 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.860220    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca71df-78d0-4f87-8801-18b32239d44c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31ca71df-78d0-4f87-8801-18b32239d44c" (UID: "31ca71df-78d0-4f87-8801-18b32239d44c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:52:17.860291 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.860247    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31ca71df-78d0-4f87-8801-18b32239d44c" (UID: "31ca71df-78d0-4f87-8801-18b32239d44c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:52:17.860291 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.860266    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31ca71df-78d0-4f87-8801-18b32239d44c" (UID: "31ca71df-78d0-4f87-8801-18b32239d44c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:52:17.862291 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.862269    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ca71df-78d0-4f87-8801-18b32239d44c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31ca71df-78d0-4f87-8801-18b32239d44c" (UID: "31ca71df-78d0-4f87-8801-18b32239d44c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:52:17.862468 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.862444    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ca71df-78d0-4f87-8801-18b32239d44c-kube-api-access-g7tnv" (OuterVolumeSpecName: "kube-api-access-g7tnv") pod "31ca71df-78d0-4f87-8801-18b32239d44c" (UID: "31ca71df-78d0-4f87-8801-18b32239d44c"). InnerVolumeSpecName "kube-api-access-g7tnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:52:17.961002 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.960960    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7tnv\" (UniqueName: \"kubernetes.io/projected/31ca71df-78d0-4f87-8801-18b32239d44c-kube-api-access-g7tnv\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:52:17.961002 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.960993    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:52:17.961002 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.961005    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31ca71df-78d0-4f87-8801-18b32239d44c-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:52:17.961002 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.961013    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ca71df-78d0-4f87-8801-18b32239d44c-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:52:17.961523 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:17.961023    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ca71df-78d0-4f87-8801-18b32239d44c-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:52:18.267266 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.267230    2581 generic.go:356] "Generic (PLEG): container finished" podID="31ca71df-78d0-4f87-8801-18b32239d44c" containerID="d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca" exitCode=0
Aug 04 18:52:18.267454 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.267289    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"
Aug 04 18:52:18.267454 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.267310    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" event={"ID":"31ca71df-78d0-4f87-8801-18b32239d44c","Type":"ContainerDied","Data":"d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca"}
Aug 04 18:52:18.267454 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.267375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77" event={"ID":"31ca71df-78d0-4f87-8801-18b32239d44c","Type":"ContainerDied","Data":"acda1b7a9ac45bdab3d27d34cc6c2186ad1ccabaf5810744db56c25819cde24e"}
Aug 04 18:52:18.267454 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.267415    2581 scope.go:117] "RemoveContainer" containerID="bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d"
Aug 04 18:52:18.269146 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.269032    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422_087dc74d-637b-4313-86dc-4082f4a72958/storage-initializer/0.log"
Aug 04 18:52:18.269266 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.269163    2581 generic.go:356] "Generic (PLEG): container finished" podID="087dc74d-637b-4313-86dc-4082f4a72958" containerID="125abab5c9bca348d76eb73ce5192f40c25b1c76201d4312bc867a0570438a6e" exitCode=1
Aug 04 18:52:18.269266 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.269211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422" event={"ID":"087dc74d-637b-4313-86dc-4082f4a72958","Type":"ContainerDied","Data":"125abab5c9bca348d76eb73ce5192f40c25b1c76201d4312bc867a0570438a6e"}
Aug 04 18:52:18.294378 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.294350    2581 scope.go:117] "RemoveContainer" containerID="d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca"
Aug 04 18:52:18.311318 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.311292    2581 scope.go:117] "RemoveContainer" containerID="467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb"
Aug 04 18:52:18.329757 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.329598    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"]
Aug 04 18:52:18.331536 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.331509    2581 scope.go:117] "RemoveContainer" containerID="bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d"
Aug 04 18:52:18.331929 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:52:18.331898    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d\": container with ID starting with bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d not found: ID does not exist" containerID="bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d"
Aug 04 18:52:18.332023 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.331943    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d"} err="failed to get container status \"bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d\": rpc error: code = NotFound desc = could not find container \"bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d\": container with ID starting with bb2e5d52b1375c0fa353da6580baef3f583426294023c7a0bd663f387634874d not found: ID does not exist"
Aug 04 18:52:18.332023 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.331970    2581 scope.go:117] "RemoveContainer" containerID="d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca"
Aug 04 18:52:18.332153 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.332026    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6b65d8ddfd-2wl77"]
Aug 04 18:52:18.332282 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:52:18.332266    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca\": container with ID starting with d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca not found: ID does not exist" containerID="d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca"
Aug 04 18:52:18.332330 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.332289    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca"} err="failed to get container status \"d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca\": rpc error: code = NotFound desc = could not find container \"d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca\": container with ID starting with d12c68fbf67fe816fc2241658c7ee1ab196c94dc2fd3c607843f3a6620a28dca not found: ID does not exist"
Aug 04 18:52:18.332330 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.332305    2581 scope.go:117] "RemoveContainer" containerID="467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb"
Aug 04 18:52:18.332590 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:52:18.332567    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb\": container with ID starting with 467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb not found: ID does not exist" containerID="467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb"
Aug 04 18:52:18.332638 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:18.332598    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb"} err="failed to get container status \"467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb\": rpc error: code = NotFound desc = could not find container \"467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb\": container with ID starting with 467017f5511cedeba6a8aa49d281241ff38dbe4d4aa171df4768a9f5c55930cb not found: ID does not exist"
Aug 04 18:52:19.214533 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:19.214490    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" path="/var/lib/kubelet/pods/31ca71df-78d0-4f87-8801-18b32239d44c/volumes"
Aug 04 18:52:19.275900 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:19.275869    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422_087dc74d-637b-4313-86dc-4082f4a72958/storage-initializer/0.log"
Aug 04 18:52:19.276121 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:19.275971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422" event={"ID":"087dc74d-637b-4313-86dc-4082f4a72958","Type":"ContainerStarted","Data":"2d97b1734f3a62a984fd700af435b61ddeeb3c2087b13c87fbee1116b4c66d69"}
Aug 04 18:52:23.296243 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:23.296213    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422_087dc74d-637b-4313-86dc-4082f4a72958/storage-initializer/1.log"
Aug 04 18:52:23.296702 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:23.296679    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422_087dc74d-637b-4313-86dc-4082f4a72958/storage-initializer/0.log"
Aug 04 18:52:23.296820 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:23.296726    2581 generic.go:356] "Generic (PLEG): container finished" podID="087dc74d-637b-4313-86dc-4082f4a72958" containerID="2d97b1734f3a62a984fd700af435b61ddeeb3c2087b13c87fbee1116b4c66d69" exitCode=1
Aug 04 18:52:23.296820 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:23.296771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422" event={"ID":"087dc74d-637b-4313-86dc-4082f4a72958","Type":"ContainerDied","Data":"2d97b1734f3a62a984fd700af435b61ddeeb3c2087b13c87fbee1116b4c66d69"}
Aug 04 18:52:23.296929 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:23.296846    2581 scope.go:117] "RemoveContainer" containerID="125abab5c9bca348d76eb73ce5192f40c25b1c76201d4312bc867a0570438a6e"
Aug 04 18:52:23.297334 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:23.297308    2581 scope.go:117] "RemoveContainer" containerID="125abab5c9bca348d76eb73ce5192f40c25b1c76201d4312bc867a0570438a6e"
Aug 04 18:52:23.308578 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:52:23.308537    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422_kserve-ci-e2e-test_087dc74d-637b-4313-86dc-4082f4a72958_0 in pod sandbox 1880bde0b777f3c33afdf07b842c224c909e2512658165d7f2d4ce6230c01a3a from index: no such id: '125abab5c9bca348d76eb73ce5192f40c25b1c76201d4312bc867a0570438a6e'" containerID="125abab5c9bca348d76eb73ce5192f40c25b1c76201d4312bc867a0570438a6e"
Aug 04 18:52:23.308716 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:52:23.308586    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422_kserve-ci-e2e-test(087dc74d-637b-4313-86dc-4082f4a72958)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422" podUID="087dc74d-637b-4313-86dc-4082f4a72958"
Aug 04 18:52:23.308716 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:23.308591    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125abab5c9bca348d76eb73ce5192f40c25b1c76201d4312bc867a0570438a6e"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422_kserve-ci-e2e-test_087dc74d-637b-4313-86dc-4082f4a72958_0 in pod sandbox 1880bde0b777f3c33afdf07b842c224c909e2512658165d7f2d4ce6230c01a3a from index: no such id: '125abab5c9bca348d76eb73ce5192f40c25b1c76201d4312bc867a0570438a6e'"
Aug 04 18:52:23.576273 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:23.576191    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"]
Aug 04 18:52:23.597323 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:23.597293    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:52:24.302002 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.301975    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422_087dc74d-637b-4313-86dc-4082f4a72958/storage-initializer/1.log"
Aug 04 18:52:24.459463 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.459436    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422_087dc74d-637b-4313-86dc-4082f4a72958/storage-initializer/1.log"
Aug 04 18:52:24.463240 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.463212    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:24.513017 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.512971    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/087dc74d-637b-4313-86dc-4082f4a72958-proxy-tls\") pod \"087dc74d-637b-4313-86dc-4082f4a72958\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") "
Aug 04 18:52:24.513235 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.513030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/087dc74d-637b-4313-86dc-4082f4a72958-kube-rbac-proxy-sar-config\") pod \"087dc74d-637b-4313-86dc-4082f4a72958\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") "
Aug 04 18:52:24.513235 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.513074    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn2w4\" (UniqueName: \"kubernetes.io/projected/087dc74d-637b-4313-86dc-4082f4a72958-kube-api-access-pn2w4\") pod \"087dc74d-637b-4313-86dc-4082f4a72958\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") "
Aug 04 18:52:24.513235 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.513112    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/087dc74d-637b-4313-86dc-4082f4a72958-kserve-provision-location\") pod \"087dc74d-637b-4313-86dc-4082f4a72958\" (UID: \"087dc74d-637b-4313-86dc-4082f4a72958\") "
Aug 04 18:52:24.513474 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.513450    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087dc74d-637b-4313-86dc-4082f4a72958-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "087dc74d-637b-4313-86dc-4082f4a72958" (UID: "087dc74d-637b-4313-86dc-4082f4a72958"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:52:24.513474 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.513456    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087dc74d-637b-4313-86dc-4082f4a72958-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "087dc74d-637b-4313-86dc-4082f4a72958" (UID: "087dc74d-637b-4313-86dc-4082f4a72958"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:52:24.515458 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.515426    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087dc74d-637b-4313-86dc-4082f4a72958-kube-api-access-pn2w4" (OuterVolumeSpecName: "kube-api-access-pn2w4") pod "087dc74d-637b-4313-86dc-4082f4a72958" (UID: "087dc74d-637b-4313-86dc-4082f4a72958"). InnerVolumeSpecName "kube-api-access-pn2w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:52:24.515569 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.515455    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/087dc74d-637b-4313-86dc-4082f4a72958-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "087dc74d-637b-4313-86dc-4082f4a72958" (UID: "087dc74d-637b-4313-86dc-4082f4a72958"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:52:24.614079 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.613979    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/087dc74d-637b-4313-86dc-4082f4a72958-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:52:24.614079 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.614011    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/087dc74d-637b-4313-86dc-4082f4a72958-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:52:24.614079 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.614027    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn2w4\" (UniqueName: \"kubernetes.io/projected/087dc74d-637b-4313-86dc-4082f4a72958-kube-api-access-pn2w4\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:52:24.614079 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.614058    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/087dc74d-637b-4313-86dc-4082f4a72958-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:52:24.664964 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.664926    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:52:24.688389 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688352    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"]
Aug 04 18:52:24.688752 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688734    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="storage-initializer"
Aug 04 18:52:24.688849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688756    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="storage-initializer"
Aug 04 18:52:24.688849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688777    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container"
Aug 04 18:52:24.688849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688788    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container"
Aug 04 18:52:24.688849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688809    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="087dc74d-637b-4313-86dc-4082f4a72958" containerName="storage-initializer"
Aug 04 18:52:24.688849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688818    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="087dc74d-637b-4313-86dc-4082f4a72958" containerName="storage-initializer"
Aug 04 18:52:24.688849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688829    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="087dc74d-637b-4313-86dc-4082f4a72958" containerName="storage-initializer"
Aug 04 18:52:24.688849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688836    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="087dc74d-637b-4313-86dc-4082f4a72958" containerName="storage-initializer"
Aug 04 18:52:24.688849 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688848    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kube-rbac-proxy"
Aug 04 18:52:24.689124 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688856    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kube-rbac-proxy"
Aug 04 18:52:24.689124 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688939    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kserve-container"
Aug 04 18:52:24.689124 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688952    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="087dc74d-637b-4313-86dc-4082f4a72958" containerName="storage-initializer"
Aug 04 18:52:24.689124 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.688961    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ca71df-78d0-4f87-8801-18b32239d44c" containerName="kube-rbac-proxy"
Aug 04 18:52:24.689301 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.689137    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="087dc74d-637b-4313-86dc-4082f4a72958" containerName="storage-initializer"
Aug 04 18:52:24.692287 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.692265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.696592 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.696565    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Aug 04 18:52:24.696730 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.696646    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Aug 04 18:52:24.704163 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.704133    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"]
Aug 04 18:52:24.714874 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.714841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njwzn\" (UniqueName: \"kubernetes.io/projected/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-api-access-njwzn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.715077 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.714881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.715077 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.714908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43f7f6bb-e01d-4050-9331-c7d88122bfbb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.715077 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.714934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.715077 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.715037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.815611 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.815571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.815823 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.815635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njwzn\" (UniqueName: \"kubernetes.io/projected/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-api-access-njwzn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.815823 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.815657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.815823 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.815680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43f7f6bb-e01d-4050-9331-c7d88122bfbb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.815823 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.815705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.816070 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:52:24.815892    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Aug 04 18:52:24.816070 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:52:24.815981    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43f7f6bb-e01d-4050-9331-c7d88122bfbb-proxy-tls podName:43f7f6bb-e01d-4050-9331-c7d88122bfbb nodeName:}" failed. No retries permitted until 2026-08-04 18:52:25.315963041 +0000 UTC m=+3170.660364188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/43f7f6bb-e01d-4050-9331-c7d88122bfbb-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" (UID: "43f7f6bb-e01d-4050-9331-c7d88122bfbb") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Aug 04 18:52:24.816204 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.816127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.816363 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.816342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.816453 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.816431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:24.829006 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:24.828968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njwzn\" (UniqueName: \"kubernetes.io/projected/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-api-access-njwzn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:25.033812 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.033773    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:52:25.235997 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.235961    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 04 18:52:25.308119 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.308029    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422_087dc74d-637b-4313-86dc-4082f4a72958/storage-initializer/1.log"
Aug 04 18:52:25.308495 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.308159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422" event={"ID":"087dc74d-637b-4313-86dc-4082f4a72958","Type":"ContainerDied","Data":"1880bde0b777f3c33afdf07b842c224c909e2512658165d7f2d4ce6230c01a3a"}
Aug 04 18:52:25.308495 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.308168    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"
Aug 04 18:52:25.308495 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.308216    2581 scope.go:117] "RemoveContainer" containerID="2d97b1734f3a62a984fd700af435b61ddeeb3c2087b13c87fbee1116b4c66d69"
Aug 04 18:52:25.320615 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.320583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43f7f6bb-e01d-4050-9331-c7d88122bfbb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:25.323411 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.323381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43f7f6bb-e01d-4050-9331-c7d88122bfbb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:25.345712 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.345672    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"]
Aug 04 18:52:25.348258 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.348225    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6789884c94-v2422"]
Aug 04 18:52:25.603165 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.603071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:25.750120 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:25.750065    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"]
Aug 04 18:52:25.752515 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:52:25.752488    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f7f6bb_e01d_4050_9331_c7d88122bfbb.slice/crio-476eba3c218f9f0a57ba37f0e98f7033112d4eb2d587d54ec3e0cd8a4e745f2b WatchSource:0}: Error finding container 476eba3c218f9f0a57ba37f0e98f7033112d4eb2d587d54ec3e0cd8a4e745f2b: Status 404 returned error can't find the container with id 476eba3c218f9f0a57ba37f0e98f7033112d4eb2d587d54ec3e0cd8a4e745f2b
Aug 04 18:52:26.316429 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:26.316385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" event={"ID":"43f7f6bb-e01d-4050-9331-c7d88122bfbb","Type":"ContainerStarted","Data":"1c99d829b6b655870fdd41898a6d20262a72d5f84defaf45d97ac5d534897917"}
Aug 04 18:52:26.316900 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:26.316459    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" event={"ID":"43f7f6bb-e01d-4050-9331-c7d88122bfbb","Type":"ContainerStarted","Data":"476eba3c218f9f0a57ba37f0e98f7033112d4eb2d587d54ec3e0cd8a4e745f2b"}
Aug 04 18:52:27.215651 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:27.215552    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="087dc74d-637b-4313-86dc-4082f4a72958" path="/var/lib/kubelet/pods/087dc74d-637b-4313-86dc-4082f4a72958/volumes"
Aug 04 18:52:27.322729 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:27.322694    2581 generic.go:356] "Generic (PLEG): container finished" podID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerID="1c99d829b6b655870fdd41898a6d20262a72d5f84defaf45d97ac5d534897917" exitCode=0
Aug 04 18:52:27.323161 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:27.322779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" event={"ID":"43f7f6bb-e01d-4050-9331-c7d88122bfbb","Type":"ContainerDied","Data":"1c99d829b6b655870fdd41898a6d20262a72d5f84defaf45d97ac5d534897917"}
Aug 04 18:52:28.330858 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:28.330819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" event={"ID":"43f7f6bb-e01d-4050-9331-c7d88122bfbb","Type":"ContainerStarted","Data":"8c7fea790b8fdd80cbde1d21766ed2446c67e913e8d2aad176a69a17201d2363"}
Aug 04 18:52:28.331277 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:28.330877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" event={"ID":"43f7f6bb-e01d-4050-9331-c7d88122bfbb","Type":"ContainerStarted","Data":"808058b39785ab52fcd58873da14088519bb96787e56ba82ba46d174d0897870"}
Aug 04 18:52:28.331277 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:28.331021    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:28.351340 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:28.351287    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podStartSLOduration=3.294973275 podStartE2EDuration="4.351271162s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.056297887s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:52:28.349574028 +0000 UTC m=+3173.693975184" watchObservedRunningTime="2026-08-04 18:52:28.351271162 +0000 UTC m=+3173.695672318"
Aug 04 18:52:29.335757 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:29.335709    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:29.337088 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:29.337057    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 04 18:52:30.340362 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:30.340319    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 04 18:52:35.235798 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:35.235751    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 04 18:52:35.345630 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:35.345599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:52:35.346359 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:35.346308    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 04 18:52:45.235810 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:45.235770    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 04 18:52:45.346281 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:45.346241    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 04 18:52:55.235546 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:55.235500    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 04 18:52:55.346615 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:52:55.346574    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 04 18:53:05.235727 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:05.235667    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 04 18:53:05.346824 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:05.346778    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 04 18:53:13.708327 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:13.708286    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:15.235670 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:15.235628    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 04 18:53:15.346539 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:15.346487    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 04 18:53:22.340167 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:22.340126    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:22.450185 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:22.450151    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:25.235836 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:25.235789    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Aug 04 18:53:25.346677 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:25.346638    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 04 18:53:35.236099 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:35.236009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:53:35.346534 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:35.346492    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 04 18:53:36.932270 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:36.932224    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"]
Aug 04 18:53:36.932664 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:36.932587    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container" containerID="cri-o://2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830" gracePeriod=30
Aug 04 18:53:36.932722 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:36.932673    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kube-rbac-proxy" containerID="cri-o://475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614" gracePeriod=30
Aug 04 18:53:36.943627 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:36.943597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:37.621441 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:37.621404    2581 generic.go:356] "Generic (PLEG): container finished" podID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerID="475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614" exitCode=2
Aug 04 18:53:37.621618 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:37.621466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" event={"ID":"1d195bcb-e83d-44f5-a550-03c0f4086de4","Type":"ContainerDied","Data":"475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614"}
Aug 04 18:53:40.229246 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.229202    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Aug 04 18:53:40.777436 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.777408    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:53:40.864541 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.864445    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d195bcb-e83d-44f5-a550-03c0f4086de4-kserve-provision-location\") pod \"1d195bcb-e83d-44f5-a550-03c0f4086de4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") "
Aug 04 18:53:40.864541 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.864523    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-rbac-proxy-sar-config\") pod \"1d195bcb-e83d-44f5-a550-03c0f4086de4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") "
Aug 04 18:53:40.864761 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.864545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d195bcb-e83d-44f5-a550-03c0f4086de4-proxy-tls\") pod \"1d195bcb-e83d-44f5-a550-03c0f4086de4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") "
Aug 04 18:53:40.864761 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.864579    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-cabundle-cert\") pod \"1d195bcb-e83d-44f5-a550-03c0f4086de4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") "
Aug 04 18:53:40.864761 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.864603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvh95\" (UniqueName: \"kubernetes.io/projected/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-api-access-gvh95\") pod \"1d195bcb-e83d-44f5-a550-03c0f4086de4\" (UID: \"1d195bcb-e83d-44f5-a550-03c0f4086de4\") "
Aug 04 18:53:40.864884 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.864811    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d195bcb-e83d-44f5-a550-03c0f4086de4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1d195bcb-e83d-44f5-a550-03c0f4086de4" (UID: "1d195bcb-e83d-44f5-a550-03c0f4086de4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:40.864955 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.864927    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d195bcb-e83d-44f5-a550-03c0f4086de4" (UID: "1d195bcb-e83d-44f5-a550-03c0f4086de4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:53:40.865101 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.864933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1d195bcb-e83d-44f5-a550-03c0f4086de4" (UID: "1d195bcb-e83d-44f5-a550-03c0f4086de4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:53:40.866891 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.866868    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d195bcb-e83d-44f5-a550-03c0f4086de4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d195bcb-e83d-44f5-a550-03c0f4086de4" (UID: "1d195bcb-e83d-44f5-a550-03c0f4086de4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:53:40.866981 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.866896    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-api-access-gvh95" (OuterVolumeSpecName: "kube-api-access-gvh95") pod "1d195bcb-e83d-44f5-a550-03c0f4086de4" (UID: "1d195bcb-e83d-44f5-a550-03c0f4086de4"). InnerVolumeSpecName "kube-api-access-gvh95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:53:40.965768 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.965717    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:53:40.965768 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.965764    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvh95\" (UniqueName: \"kubernetes.io/projected/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-api-access-gvh95\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:53:40.965768 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.965776    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d195bcb-e83d-44f5-a550-03c0f4086de4-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:53:40.965768 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.965785    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d195bcb-e83d-44f5-a550-03c0f4086de4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:53:40.966082 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:40.965794    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d195bcb-e83d-44f5-a550-03c0f4086de4-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:53:41.210230 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.210147    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Aug 04 18:53:41.638026 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.637986    2581 generic.go:356] "Generic (PLEG): container finished" podID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerID="2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830" exitCode=0
Aug 04 18:53:41.638463 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.638036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" event={"ID":"1d195bcb-e83d-44f5-a550-03c0f4086de4","Type":"ContainerDied","Data":"2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830"}
Aug 04 18:53:41.638463 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.638069    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"
Aug 04 18:53:41.638463 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.638129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4" event={"ID":"1d195bcb-e83d-44f5-a550-03c0f4086de4","Type":"ContainerDied","Data":"0530ad92f183d07ff9e935f5148ee1603b9998f26f755eeaf3a8657045cc9bc4"}
Aug 04 18:53:41.638463 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.638167    2581 scope.go:117] "RemoveContainer" containerID="475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614"
Aug 04 18:53:41.647169 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.646970    2581 scope.go:117] "RemoveContainer" containerID="2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830"
Aug 04 18:53:41.655560 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.655534    2581 scope.go:117] "RemoveContainer" containerID="0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0"
Aug 04 18:53:41.660106 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.660077    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"]
Aug 04 18:53:41.665111 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.664881    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c7rx4"]
Aug 04 18:53:41.665111 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.665095    2581 scope.go:117] "RemoveContainer" containerID="475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614"
Aug 04 18:53:41.665479 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:53:41.665460    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614\": container with ID starting with 475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614 not found: ID does not exist" containerID="475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614"
Aug 04 18:53:41.665546 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.665491    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614"} err="failed to get container status \"475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614\": rpc error: code = NotFound desc = could not find container \"475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614\": container with ID starting with 475f58eaee6c2a22bc577488e75bb83e602a3a7535f27d33363078f7c5797614 not found: ID does not exist"
Aug 04 18:53:41.665546 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.665517    2581 scope.go:117] "RemoveContainer" containerID="2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830"
Aug 04 18:53:41.665790 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:53:41.665770    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830\": container with ID starting with 2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830 not found: ID does not exist" containerID="2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830"
Aug 04 18:53:41.665832 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.665801    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830"} err="failed to get container status \"2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830\": rpc error: code = NotFound desc = could not find container \"2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830\": container with ID starting with 2c0469d19bcbf87710eac49a9fc881b631eac05e6a7fd17e478b54481ef03830 not found: ID does not exist"
Aug 04 18:53:41.665832 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.665821    2581 scope.go:117] "RemoveContainer" containerID="0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0"
Aug 04 18:53:41.666420 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:53:41.666395    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0\": container with ID starting with 0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0 not found: ID does not exist" containerID="0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0"
Aug 04 18:53:41.666472 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:41.666426    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0"} err="failed to get container status \"0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0\": rpc error: code = NotFound desc = could not find container \"0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0\": container with ID starting with 0cb2d18d608c9d67c57ee0d6d1d326657afe5c3b516aeaf9f87bc51178aa5da0 not found: ID does not exist"
Aug 04 18:53:42.102753 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:42.102719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:42.296630 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:42.296597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:43.214236 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:43.214206    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" path="/var/lib/kubelet/pods/1d195bcb-e83d-44f5-a550-03c0f4086de4/volumes"
Aug 04 18:53:51.214158 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:51.214127    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:53:54.790290 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:54.790254    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"]
Aug 04 18:53:54.790767 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:54.790502    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container" containerID="cri-o://808058b39785ab52fcd58873da14088519bb96787e56ba82ba46d174d0897870" gracePeriod=30
Aug 04 18:53:54.790767 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:54.790553    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kube-rbac-proxy" containerID="cri-o://8c7fea790b8fdd80cbde1d21766ed2446c67e913e8d2aad176a69a17201d2363" gracePeriod=30
Aug 04 18:53:54.817233 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:54.817201    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:55.340453 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.340409    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Aug 04 18:53:55.697137 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.697056    2581 generic.go:356] "Generic (PLEG): container finished" podID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerID="8c7fea790b8fdd80cbde1d21766ed2446c67e913e8d2aad176a69a17201d2363" exitCode=2
Aug 04 18:53:55.697137 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.697083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" event={"ID":"43f7f6bb-e01d-4050-9331-c7d88122bfbb","Type":"ContainerDied","Data":"8c7fea790b8fdd80cbde1d21766ed2446c67e913e8d2aad176a69a17201d2363"}
Aug 04 18:53:55.838210 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.838173    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"]
Aug 04 18:53:55.838600 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.838495    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container"
Aug 04 18:53:55.838600 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.838507    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container"
Aug 04 18:53:55.838600 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.838516    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kube-rbac-proxy"
Aug 04 18:53:55.838600 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.838522    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kube-rbac-proxy"
Aug 04 18:53:55.838600 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.838534    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="storage-initializer"
Aug 04 18:53:55.838600 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.838540    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="storage-initializer"
Aug 04 18:53:55.838600 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.838600    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kube-rbac-proxy"
Aug 04 18:53:55.838814 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.838607    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d195bcb-e83d-44f5-a550-03c0f4086de4" containerName="kserve-container"
Aug 04 18:53:55.846472 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.846438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:55.849454 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.849425    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Aug 04 18:53:55.850431 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.850411    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Aug 04 18:53:55.856837 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.856806    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"]
Aug 04 18:53:55.880815 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:55.880779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:56.003669 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.003582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285d43c9-a544-4f32-8b21-ade1b3fba665-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.003669 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.003626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vstvd\" (UniqueName: \"kubernetes.io/projected/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-api-access-vstvd\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.003669 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.003656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285d43c9-a544-4f32-8b21-ade1b3fba665-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.003894 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.003741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.072731 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.072694    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:53:56.104213 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.104175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.104445 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.104249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285d43c9-a544-4f32-8b21-ade1b3fba665-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.104445 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.104276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vstvd\" (UniqueName: \"kubernetes.io/projected/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-api-access-vstvd\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.104445 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.104301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285d43c9-a544-4f32-8b21-ade1b3fba665-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.104736 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.104716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285d43c9-a544-4f32-8b21-ade1b3fba665-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.104927 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.104906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.106952 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.106930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285d43c9-a544-4f32-8b21-ade1b3fba665-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.119224 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.119195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vstvd\" (UniqueName: \"kubernetes.io/projected/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-api-access-vstvd\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.158419 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.158365    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:53:56.324973 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.324937    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"]
Aug 04 18:53:56.329032 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:53:56.328995    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285d43c9_a544_4f32_8b21_ade1b3fba665.slice/crio-705c64eee291c604bd12b9e7f578441c9c5229fc32183f8edf1761cbe69e8dce WatchSource:0}: Error finding container 705c64eee291c604bd12b9e7f578441c9c5229fc32183f8edf1761cbe69e8dce: Status 404 returned error can't find the container with id 705c64eee291c604bd12b9e7f578441c9c5229fc32183f8edf1761cbe69e8dce
Aug 04 18:53:56.703110 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.703014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh" event={"ID":"285d43c9-a544-4f32-8b21-ade1b3fba665","Type":"ContainerStarted","Data":"01a1f662a987ac0c439a7926947f352e42587721eb22af19428e639d608e8e08"}
Aug 04 18:53:56.703110 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:56.703101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh" event={"ID":"285d43c9-a544-4f32-8b21-ade1b3fba665","Type":"ContainerStarted","Data":"705c64eee291c604bd12b9e7f578441c9c5229fc32183f8edf1761cbe69e8dce"}
Aug 04 18:53:59.717299 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.717264    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh_285d43c9-a544-4f32-8b21-ade1b3fba665/storage-initializer/0.log"
Aug 04 18:53:59.717740 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.717317    2581 generic.go:356] "Generic (PLEG): container finished" podID="285d43c9-a544-4f32-8b21-ade1b3fba665" containerID="01a1f662a987ac0c439a7926947f352e42587721eb22af19428e639d608e8e08" exitCode=1
Aug 04 18:53:59.717740 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.717392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh" event={"ID":"285d43c9-a544-4f32-8b21-ade1b3fba665","Type":"ContainerDied","Data":"01a1f662a987ac0c439a7926947f352e42587721eb22af19428e639d608e8e08"}
Aug 04 18:53:59.720036 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.719938    2581 generic.go:356] "Generic (PLEG): container finished" podID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerID="808058b39785ab52fcd58873da14088519bb96787e56ba82ba46d174d0897870" exitCode=0
Aug 04 18:53:59.720036 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.720014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" event={"ID":"43f7f6bb-e01d-4050-9331-c7d88122bfbb","Type":"ContainerDied","Data":"808058b39785ab52fcd58873da14088519bb96787e56ba82ba46d174d0897870"}
Aug 04 18:53:59.766023 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.765995    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:53:59.935340 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.935241    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kserve-provision-location\") pod \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") "
Aug 04 18:53:59.935340 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.935283    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njwzn\" (UniqueName: \"kubernetes.io/projected/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-api-access-njwzn\") pod \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") "
Aug 04 18:53:59.935340 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.935318    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-rbac-proxy-sar-config\") pod \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") "
Aug 04 18:53:59.935646 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.935357    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-cabundle-cert\") pod \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") "
Aug 04 18:53:59.935646 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.935420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43f7f6bb-e01d-4050-9331-c7d88122bfbb-proxy-tls\") pod \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\" (UID: \"43f7f6bb-e01d-4050-9331-c7d88122bfbb\") "
Aug 04 18:53:59.935744 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.935655    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43f7f6bb-e01d-4050-9331-c7d88122bfbb" (UID: "43f7f6bb-e01d-4050-9331-c7d88122bfbb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:59.935744 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.935731    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43f7f6bb-e01d-4050-9331-c7d88122bfbb" (UID: "43f7f6bb-e01d-4050-9331-c7d88122bfbb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:53:59.935856 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.935793    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43f7f6bb-e01d-4050-9331-c7d88122bfbb" (UID: "43f7f6bb-e01d-4050-9331-c7d88122bfbb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:53:59.937772 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.937743    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f7f6bb-e01d-4050-9331-c7d88122bfbb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43f7f6bb-e01d-4050-9331-c7d88122bfbb" (UID: "43f7f6bb-e01d-4050-9331-c7d88122bfbb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:53:59.937868 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:53:59.937806    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-api-access-njwzn" (OuterVolumeSpecName: "kube-api-access-njwzn") pod "43f7f6bb-e01d-4050-9331-c7d88122bfbb" (UID: "43f7f6bb-e01d-4050-9331-c7d88122bfbb"). InnerVolumeSpecName "kube-api-access-njwzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:54:00.036417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.036374    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43f7f6bb-e01d-4050-9331-c7d88122bfbb-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.036417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.036407    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.036417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.036417    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njwzn\" (UniqueName: \"kubernetes.io/projected/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-api-access-njwzn\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.036417 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.036427    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.036692 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.036436    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43f7f6bb-e01d-4050-9331-c7d88122bfbb-cabundle-cert\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:54:00.725638 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.725608    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh_285d43c9-a544-4f32-8b21-ade1b3fba665/storage-initializer/0.log"
Aug 04 18:54:00.725996 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.725716    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh" event={"ID":"285d43c9-a544-4f32-8b21-ade1b3fba665","Type":"ContainerStarted","Data":"e3f53d46ecbce3ace5a3c707bc96c4342c6f08579a338766ff00231680a70ffe"}
Aug 04 18:54:00.727566 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.727542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq" event={"ID":"43f7f6bb-e01d-4050-9331-c7d88122bfbb","Type":"ContainerDied","Data":"476eba3c218f9f0a57ba37f0e98f7033112d4eb2d587d54ec3e0cd8a4e745f2b"}
Aug 04 18:54:00.727683 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.727575    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"
Aug 04 18:54:00.727683 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.727605    2581 scope.go:117] "RemoveContainer" containerID="8c7fea790b8fdd80cbde1d21766ed2446c67e913e8d2aad176a69a17201d2363"
Aug 04 18:54:00.736822 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.736797    2581 scope.go:117] "RemoveContainer" containerID="808058b39785ab52fcd58873da14088519bb96787e56ba82ba46d174d0897870"
Aug 04 18:54:00.745104 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.745085    2581 scope.go:117] "RemoveContainer" containerID="1c99d829b6b655870fdd41898a6d20262a72d5f84defaf45d97ac5d534897917"
Aug 04 18:54:00.775419 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.775375    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"]
Aug 04 18:54:00.781796 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:00.781759    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7d56db8dc7-ml7xq"]
Aug 04 18:54:01.215573 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:01.215536    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" path="/var/lib/kubelet/pods/43f7f6bb-e01d-4050-9331-c7d88122bfbb/volumes"
Aug 04 18:54:03.742817 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:03.742789    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh_285d43c9-a544-4f32-8b21-ade1b3fba665/storage-initializer/1.log"
Aug 04 18:54:03.743229 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:03.743188    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh_285d43c9-a544-4f32-8b21-ade1b3fba665/storage-initializer/0.log"
Aug 04 18:54:03.743229 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:03.743217    2581 generic.go:356] "Generic (PLEG): container finished" podID="285d43c9-a544-4f32-8b21-ade1b3fba665" containerID="e3f53d46ecbce3ace5a3c707bc96c4342c6f08579a338766ff00231680a70ffe" exitCode=1
Aug 04 18:54:03.743341 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:03.743262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh" event={"ID":"285d43c9-a544-4f32-8b21-ade1b3fba665","Type":"ContainerDied","Data":"e3f53d46ecbce3ace5a3c707bc96c4342c6f08579a338766ff00231680a70ffe"}
Aug 04 18:54:03.743341 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:03.743315    2581 scope.go:117] "RemoveContainer" containerID="01a1f662a987ac0c439a7926947f352e42587721eb22af19428e639d608e8e08"
Aug 04 18:54:03.743704 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:03.743686    2581 scope.go:117] "RemoveContainer" containerID="01a1f662a987ac0c439a7926947f352e42587721eb22af19428e639d608e8e08"
Aug 04 18:54:03.753191 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:54:03.753157    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh_kserve-ci-e2e-test(285d43c9-a544-4f32-8b21-ade1b3fba665)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh" podUID="285d43c9-a544-4f32-8b21-ade1b3fba665"
Aug 04 18:54:03.753191 ip-10-0-129-212 kubenswrapper[2581]: E0804 18:54:03.753164    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh_kserve-ci-e2e-test_285d43c9-a544-4f32-8b21-ade1b3fba665_0 in pod sandbox 705c64eee291c604bd12b9e7f578441c9c5229fc32183f8edf1761cbe69e8dce from index: no such id: '01a1f662a987ac0c439a7926947f352e42587721eb22af19428e639d608e8e08'" containerID="01a1f662a987ac0c439a7926947f352e42587721eb22af19428e639d608e8e08"
Aug 04 18:54:03.753350 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:03.753208    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a1f662a987ac0c439a7926947f352e42587721eb22af19428e639d608e8e08"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh_kserve-ci-e2e-test_285d43c9-a544-4f32-8b21-ade1b3fba665_0 in pod sandbox 705c64eee291c604bd12b9e7f578441c9c5229fc32183f8edf1761cbe69e8dce from index: no such id: '01a1f662a987ac0c439a7926947f352e42587721eb22af19428e639d608e8e08'"
Aug 04 18:54:04.749361 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:04.749333    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh_285d43c9-a544-4f32-8b21-ade1b3fba665/storage-initializer/1.log"
Aug 04 18:54:05.884801 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:05.884754    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"]
Aug 04 18:54:05.906419 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:05.906365    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:06.020067 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.020020    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh_285d43c9-a544-4f32-8b21-ade1b3fba665/storage-initializer/1.log"
Aug 04 18:54:06.022636 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.022613    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:54:06.191556 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.191460    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vstvd\" (UniqueName: \"kubernetes.io/projected/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-api-access-vstvd\") pod \"285d43c9-a544-4f32-8b21-ade1b3fba665\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") "
Aug 04 18:54:06.191556 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.191505    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285d43c9-a544-4f32-8b21-ade1b3fba665-proxy-tls\") pod \"285d43c9-a544-4f32-8b21-ade1b3fba665\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") "
Aug 04 18:54:06.191556 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.191552    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285d43c9-a544-4f32-8b21-ade1b3fba665-kserve-provision-location\") pod \"285d43c9-a544-4f32-8b21-ade1b3fba665\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") "
Aug 04 18:54:06.191859 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.191594    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-rbac-proxy-sar-config\") pod \"285d43c9-a544-4f32-8b21-ade1b3fba665\" (UID: \"285d43c9-a544-4f32-8b21-ade1b3fba665\") "
Aug 04 18:54:06.191859 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.191827    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285d43c9-a544-4f32-8b21-ade1b3fba665-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "285d43c9-a544-4f32-8b21-ade1b3fba665" (UID: "285d43c9-a544-4f32-8b21-ade1b3fba665"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:54:06.192028 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.192005    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "285d43c9-a544-4f32-8b21-ade1b3fba665" (UID: "285d43c9-a544-4f32-8b21-ade1b3fba665"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:54:06.194348 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.194313    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-api-access-vstvd" (OuterVolumeSpecName: "kube-api-access-vstvd") pod "285d43c9-a544-4f32-8b21-ade1b3fba665" (UID: "285d43c9-a544-4f32-8b21-ade1b3fba665"). InnerVolumeSpecName "kube-api-access-vstvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:54:06.194456 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.194314    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285d43c9-a544-4f32-8b21-ade1b3fba665-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "285d43c9-a544-4f32-8b21-ade1b3fba665" (UID: "285d43c9-a544-4f32-8b21-ade1b3fba665"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:54:06.293111 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.293075    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vstvd\" (UniqueName: \"kubernetes.io/projected/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-api-access-vstvd\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:54:06.293111 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.293108    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285d43c9-a544-4f32-8b21-ade1b3fba665-proxy-tls\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:54:06.293111 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.293118    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285d43c9-a544-4f32-8b21-ade1b3fba665-kserve-provision-location\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:54:06.293413 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.293128    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285d43c9-a544-4f32-8b21-ade1b3fba665-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-212.ec2.internal\" DevicePath \"\""
Aug 04 18:54:06.761102 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.761070    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh_285d43c9-a544-4f32-8b21-ade1b3fba665/storage-initializer/1.log"
Aug 04 18:54:06.761282 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.761136    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh" event={"ID":"285d43c9-a544-4f32-8b21-ade1b3fba665","Type":"ContainerDied","Data":"705c64eee291c604bd12b9e7f578441c9c5229fc32183f8edf1761cbe69e8dce"}
Aug 04 18:54:06.761282 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.761179    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"
Aug 04 18:54:06.761282 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.761195    2581 scope.go:117] "RemoveContainer" containerID="e3f53d46ecbce3ace5a3c707bc96c4342c6f08579a338766ff00231680a70ffe"
Aug 04 18:54:06.811026 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.810979    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"]
Aug 04 18:54:06.812713 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:06.812679    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6b57bbfcd7-p9vxh"]
Aug 04 18:54:07.215172 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:07.215140    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285d43c9-a544-4f32-8b21-ade1b3fba665" path="/var/lib/kubelet/pods/285d43c9-a544-4f32-8b21-ade1b3fba665/volumes"
Aug 04 18:54:18.848664 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:18.848629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:54:35.380502 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:35.380466    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:54:35.382409 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:35.382384    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:54:35.387805 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:35.387786    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:54:35.389704 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:54:35.389687    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:55:14.025892 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:55:14.025847    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:55:20.519731 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:55:20.519699    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:55:21.036913 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:55:21.036865    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:55:56.342182 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:55:56.342146    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:55:56.536863 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:55:56.536832    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:55:56.616163 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:55:56.616074    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:56:01.330484 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:01.330449    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ksxcz_8626a82d-8bbc-4e96-bd2d-bf7a1ec40786/global-pull-secret-syncer/0.log"
Aug 04 18:56:01.608233 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:01.608154    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log"
Aug 04 18:56:01.765669 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:01.765640    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-212.ec2.internal_f98a59467503bad43221398828496826/haproxy/0.log"
Aug 04 18:56:02.751778 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:02.751744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:56:03.133904 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:03.133872    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:56:05.832823 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:05.832785    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-wvj9z_2335dcc1-9d52-483e-b1c6-1e58887ec4ca/cluster-monitoring-operator/0.log"
Aug 04 18:56:06.236416 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:06.236331    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48q26_c49f0a35-6d58-4699-94c2-b2ad4baf96ee/node-exporter/0.log"
Aug 04 18:56:06.283382 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:06.283356    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48q26_c49f0a35-6d58-4699-94c2-b2ad4baf96ee/kube-rbac-proxy/0.log"
Aug 04 18:56:06.333112 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:06.333082    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48q26_c49f0a35-6d58-4699-94c2-b2ad4baf96ee/init-textfile/0.log"
Aug 04 18:56:08.901797 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:08.901769    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-tlnn6_37559df0-a4d3-48ac-87be-21de5499def8/networking-console-plugin/0.log"
Aug 04 18:56:09.400630 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.400601    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/1.log"
Aug 04 18:56:09.405706 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.405674    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-xgggh_3a326eea-6141-4dca-a686-a9fa0dcdbaf6/console-operator/2.log"
Aug 04 18:56:09.572931 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.572887    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"]
Aug 04 18:56:09.573221 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573205    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="storage-initializer"
Aug 04 18:56:09.573281 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573226    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="storage-initializer"
Aug 04 18:56:09.573281 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573244    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285d43c9-a544-4f32-8b21-ade1b3fba665" containerName="storage-initializer"
Aug 04 18:56:09.573281 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573253    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="285d43c9-a544-4f32-8b21-ade1b3fba665" containerName="storage-initializer"
Aug 04 18:56:09.573281 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573268    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kube-rbac-proxy"
Aug 04 18:56:09.573281 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573274    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kube-rbac-proxy"
Aug 04 18:56:09.573501 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573301    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container"
Aug 04 18:56:09.573501 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573308    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container"
Aug 04 18:56:09.573501 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573373    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kserve-container"
Aug 04 18:56:09.573501 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573383    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="285d43c9-a544-4f32-8b21-ade1b3fba665" containerName="storage-initializer"
Aug 04 18:56:09.573501 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573390    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="285d43c9-a544-4f32-8b21-ade1b3fba665" containerName="storage-initializer"
Aug 04 18:56:09.573501 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.573398    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f7f6bb-e01d-4050-9331-c7d88122bfbb" containerName="kube-rbac-proxy"
Aug 04 18:56:09.576380 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.576363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.578957 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.578932    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vp9hh\"/\"kube-root-ca.crt\""
Aug 04 18:56:09.579105 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.578977    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vp9hh\"/\"openshift-service-ca.crt\""
Aug 04 18:56:09.579778 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.579761    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vp9hh\"/\"default-dockercfg-vqdkp\""
Aug 04 18:56:09.590262 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.590229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"]
Aug 04 18:56:09.613449 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.613415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-lib-modules\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.613449 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.613454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-podres\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.613658 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.613483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-proc\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.613658 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.613509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8fj5\" (UniqueName: \"kubernetes.io/projected/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-kube-api-access-h8fj5\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.613658 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.613534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-sys\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.714138 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.714035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-proc\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.714138 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.714112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fj5\" (UniqueName: \"kubernetes.io/projected/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-kube-api-access-h8fj5\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.714316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.714155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-sys\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.714316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.714160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-proc\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.714316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.714220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-sys\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.714316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.714234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-lib-modules\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.714316 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.714268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-podres\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.714514 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.714388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-podres\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.714514 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.714403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-lib-modules\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.723775 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.723750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8fj5\" (UniqueName: \"kubernetes.io/projected/7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b-kube-api-access-h8fj5\") pod \"perf-node-gather-daemonset-sfqnb\" (UID: \"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:09.887657 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:09.887619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:10.028783 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:10.028749    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"]
Aug 04 18:56:10.032543 ip-10-0-129-212 kubenswrapper[2581]: W0804 18:56:10.032512    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7f4f8e8f_1bda_4044_8ffa_f3fd3f3ee76b.slice/crio-8e6775a950f451f05a6c7cee33a1a515a6cb5416c4afd50dcb32d34365224973 WatchSource:0}: Error finding container 8e6775a950f451f05a6c7cee33a1a515a6cb5416c4afd50dcb32d34365224973: Status 404 returned error can't find the container with id 8e6775a950f451f05a6c7cee33a1a515a6cb5416c4afd50dcb32d34365224973
Aug 04 18:56:10.242808 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:10.242706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb" event={"ID":"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b","Type":"ContainerStarted","Data":"cba948ded50498040d098d20278fadb209168328e300f65dca121a181be754b1"}
Aug 04 18:56:10.242808 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:10.242763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb" event={"ID":"7f4f8e8f-1bda-4044-8ffa-f3fd3f3ee76b","Type":"ContainerStarted","Data":"8e6775a950f451f05a6c7cee33a1a515a6cb5416c4afd50dcb32d34365224973"}
Aug 04 18:56:10.243073 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:10.242844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:10.265108 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:10.265023    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb" podStartSLOduration=1.265007059 podStartE2EDuration="1.265007059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:56:10.263515098 +0000 UTC m=+3395.607916256" watchObservedRunningTime="2026-08-04 18:56:10.265007059 +0000 UTC m=+3395.609408216"
Aug 04 18:56:10.633585 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:10.633553    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-qk8jv_d5277a01-9455-4067-9d90-067385c2116b/volume-data-source-validator/0.log"
Aug 04 18:56:11.644655 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:11.644622    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5x45c_22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7/dns/0.log"
Aug 04 18:56:11.672914 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:11.672885    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5x45c_22f5bf10-2cd9-45c2-b2f0-7f29b134b6e7/kube-rbac-proxy/0.log"
Aug 04 18:56:11.914242 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:11.914166    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nnr9j_883287f9-439b-406f-92be-57c88b121f72/dns-node-resolver/0.log"
Aug 04 18:56:12.575471 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:12.575431    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n9kl_41812be4-3b11-4893-a17b-7a67d8938e03/konnectivity-agent/0.log" limit=1024
Aug 04 18:56:12.603317 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:12.603266    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gwkqx_6516e99e-3bc1-46a1-ad86-2f72e109c596/node-ca/0.log"
Aug 04 18:56:13.428697 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:13.428669    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-785c67bddb-zlhb4_db104465-bd2b-4963-b19c-26f2b8438865/router/0.log"
Aug 04 18:56:13.839306 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:13.839272    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8brzb_d80c332a-acaa-4ecb-b197-e69f0c4116d5/serve-healthcheck-canary/0.log"
Aug 04 18:56:14.361334 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:14.361297    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-zxjxn_7e8f66e5-a4db-45ee-877a-de0246618be7/insights-operator/0.log"
Aug 04 18:56:14.363006 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:14.362988    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-zxjxn_7e8f66e5-a4db-45ee-877a-de0246618be7/insights-operator/1.log"
Aug 04 18:56:14.473199 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:14.473171    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f767j_87611d06-12e3-4b27-8ce5-a156441f8b9e/kube-rbac-proxy/0.log"
Aug 04 18:56:14.499230 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:14.499208    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f767j_87611d06-12e3-4b27-8ce5-a156441f8b9e/exporter/0.log"
Aug 04 18:56:14.525879 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:14.525852    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f767j_87611d06-12e3-4b27-8ce5-a156441f8b9e/extractor/0.log"
Aug 04 18:56:16.257073 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:16.257012    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-sfqnb"
Aug 04 18:56:17.031700 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:17.031663    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5f9d7f69b6-p7zxf_6251f072-33a3-47f5-b8da-33fe89b5b1e5/manager/0.log"
Aug 04 18:56:17.941888 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:17.941860    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-smfnx_407cd864-f36b-4ec7-a59b-64fb9cb8b1d9/seaweedfs/0.log"
Aug 04 18:56:23.399922 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:23.399888    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-xftn9_0e7b1bac-ba42-4f89-bc21-b06342fec05c/kube-storage-version-migrator-operator/1.log"
Aug 04 18:56:23.400636 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:23.400620    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-xftn9_0e7b1bac-ba42-4f89-bc21-b06342fec05c/kube-storage-version-migrator-operator/0.log"
Aug 04 18:56:24.610223 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:24.610193    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c28cc_442c48c7-0e14-4940-b1e1-ef2a4b053953/kube-multus-additional-cni-plugins/0.log"
Aug 04 18:56:24.644936 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:24.644908    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c28cc_442c48c7-0e14-4940-b1e1-ef2a4b053953/egress-router-binary-copy/0.log"
Aug 04 18:56:24.685351 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:24.685311    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c28cc_442c48c7-0e14-4940-b1e1-ef2a4b053953/cni-plugins/0.log"
Aug 04 18:56:24.713022 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:24.712996    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c28cc_442c48c7-0e14-4940-b1e1-ef2a4b053953/bond-cni-plugin/0.log"
Aug 04 18:56:24.741568 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:24.741534    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c28cc_442c48c7-0e14-4940-b1e1-ef2a4b053953/routeoverride-cni/0.log"
Aug 04 18:56:24.779130 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:24.779098    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c28cc_442c48c7-0e14-4940-b1e1-ef2a4b053953/whereabouts-cni-bincopy/0.log"
Aug 04 18:56:24.807901 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:24.807877    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c28cc_442c48c7-0e14-4940-b1e1-ef2a4b053953/whereabouts-cni/0.log"
Aug 04 18:56:25.108303 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:25.108271    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-crxfm_9faf7e40-0a3c-472f-8bbd-2897f7bcb23e/kube-multus/0.log"
Aug 04 18:56:25.299243 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:25.299206    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gdpj9_56e86da9-f3b0-4819-872b-8f4944af85e8/network-metrics-daemon/0.log"
Aug 04 18:56:25.329251 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:25.329225    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gdpj9_56e86da9-f3b0-4819-872b-8f4944af85e8/kube-rbac-proxy/0.log"
Aug 04 18:56:26.401916 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:26.401866    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-controller/0.log"
Aug 04 18:56:26.426177 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:26.426140    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/1.log"
Aug 04 18:56:26.426177 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:26.426168    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovn-acl-logging/0.log"
Aug 04 18:56:26.455233 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:26.455201    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/kube-rbac-proxy-node/0.log"
Aug 04 18:56:26.500833 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:26.500802    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/kube-rbac-proxy-ovn-metrics/0.log"
Aug 04 18:56:26.555932 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:26.555904    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/northd/0.log"
Aug 04 18:56:26.612818 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:26.612787    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/nbdb/0.log"
Aug 04 18:56:26.675267 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:26.675188    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/sbdb/0.log"
Aug 04 18:56:26.727439 ip-10-0-129-212 kubenswrapper[2581]: I0804 18:56:26.727403    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2wfn_feb74e0b-0f63-47a7-8853-85ee79a22f73/ovnkube-controller/0.log"